Run the Judge

Once the system has run and relevant artefacts are captured, the judge assesses these artefacts against the evaluation design from earlier.

Likewise, we have to manage reliability and observability to ensure that the judgements are trustworthy.

---

Managing evaluation reliability

We seek to judge the same system behaviour as consistently as possible across repeated judgements, through minimising avoidable variation and managing inherent randomness.

Here are some best practices to achieve this:

Minimising avoidable variation

1. Fix whatever can drift in your chosen judge(s).
   • Rule-based: metric implementation, thresholds, dependencies.
   • Model-based: judge model and version, prompt, rubric, inference settings.
   • Human: rubric, instructions, examples, reviewer training.
2. Control procedural effects during execution. Avoid introducing differences through how cases are presented, assigned, or contextualised. For example:
   • Pairwise comparisons: Swap response order to reduce position effects.
   • Human reviewers: Balance case assignment to reduce reviewer effects.
   • Reference: Show judges the same information to avoid differences in what they see.

Managing inherent randomness

1. Repeat judgements when needed, and keep them independent. Some evaluation designs carry inherent randomness, so repeating them reduces noise. Keep each repeat independent:
   • Human reviewers: Judge independently before discussion or adjudication.
   • Model-based judges: Run each repeat fresh, without exposing prior judgements.

Notes: LLM-based evaluation designs

• Reduce sampling noise as much as possible:
  • Use a low temperature (often 0).
  • Fix seed where the API offers one, to make repeats more reproducible.
  • Keep other sampling parameters fixed.
• Watch for truncation: Record stop reasons and token usage so judgements cut off by context or output limits are not silently treated as valid results.
• Handle caching deliberately: Disable cached judge responses when repeated calls are intended to measure sampling variation; keep caching consistent otherwise.

Don't over-repeat judgements

Repeating judgements carries the same near-linear cost as repeating runs, so scale the count to the stakes and the dimension × resource trade-off.

Managing evaluation observability

Explainability matters most for types of judges that might have an opaque methodology to cast judgements. Each has their own practices for making judgements more explainable:

Judge type	How to make the score explainable
Human	Ask reviewers to record a brief rationale alongside each judgement, with clarifications where the call is close.
Model-based	Understand the model behind each score: which model it is, and how transparent it is.

Notes: Improving observability in LLM judges

LLM judges can be especially opaque. Some practices to improve observability:

• Ask for a justification, not just a verdict. Require a brief reason in the output schema that cites the evidence used.
• Ask for a confidence estimate. This can help surface uncertainty, but treat self-reported confidence with caution because it may not be well calibrated.
• Capture structured reasoning signals for complex judgements. Record concise intermediate conclusions or criterion-level assessments rather than relying only on the final score.

For example, suppose an LLM judge is checking whether a GrantsAssist response correctly explains a user's eligibility.

Instead of returning only:

{
  "verdict": "incorrect"
}

ask it to return:

{
  "verdict": "incorrect",
  "justification": "The response states that applicants must be Singapore citizens, but the supplied eligibility criteria also allow permanent residents."
}

Don't overload the judge

Keep extra output requirements to a minimum. Asking for too much can reduce the quality of the verdict itself.

Executing a full evaluation run

With an understanding of how to ensure reliable judgements and capture the necessary signals, you are ready to execute a full evaluation run.

At this stage, execution largely involves putting your evaluation design into practice. Refer to Types of Judges, particularly the guidance on judge calibration, for additional considerations when running your judges.

Below is a full worked example of an evaluation run to showcase how the best practices highlighted above can be applied.

Worked example: A full evaluation run

The team judges GrantsAssist's roughly 5,000 captured artefacts with an LLM judge.

Reliability and observability, baked into how the judge is called:

• Pinned: Model and version, rubric, and temperature=0 are fixed, so the same artefact returns the same verdict.
• Independent repeats: Each of the 3 calibration repeats is a fresh call, so no earlier verdict leaks in.
• Justification in the schema: Every verdict carries a one-line reason, so a surprising call can be traced back.

from anthropic import Anthropic

client = Anthropic()

# Pin the judge: model and version, rubric, and sampling are fixed for the whole
# run, so re-judging the same artefact returns the same verdict.
JUDGE_MODEL = "claude-opus-4-8-20260115"  # pinned version, not a floating alias
RUBRIC = (
    "Grade whether the GrantsAssist reply states the applicant's eligibility "
    "correctly, given the official criteria. Reply only as JSON: "
    '{"verdict": "correct" | "incorrect", '
    '"justification": "<one line citing the evidence you used>"}'
)

def judge(artefact):
    response = client.messages.create(
        model=JUDGE_MODEL,
        system=RUBRIC,      # fixed rubric
        temperature=0,      # no sampling: same input, same verdict
        max_tokens=256,
        messages=[{"role": "user", "content": artefact}],
    )
    # Capture the why: the schema carries a justification, so a surprising
    # verdict can be traced back to what drove it.
    return response.content[0].text

# Keep repeats independent: on the 200-case calibration slice each repeat is a
# fresh call, so an earlier verdict never leaks into a later one.
verdicts = [judge(artefact) for _ in range(3)]

Each call returns the verdict together with the justification that drove it:

{
  "verdict": "incorrect",
  "justification": "The reply says applicants must be Singapore citizens, but the official criteria also allow permanent residents."
}

Across the 3 calibration repeats the verdict stayed incorrect; only the justification wording drifted, which is the residual noise the repeats measure.

Before the run

• Dry-ran a handful of artefacts end to end, catching schema and parsing errors before launch.
• Estimated the run at ~5,000 calls (~$0.02 and ~8 s each), which stayed within budget.

During the run

• Retried rate-limited calls with backoff, so none was scored as a failure.
• Checkpointed verdicts as they landed, so a crash resumes from the last checkpoint.
• Halted the run if invalid or unparseable judgements exceeded 5%; they peaked at 1.2%, so it ran to completion.

Outcome: Every artefact ends with a verdict, or a recorded reason it has none.

---

Before you head off

Every test case has been run: a judgement, or a recorded reason it has none (e.g. timeout, invalid judgement)
Failed and invalid rows are kept, not deleted

Once the system has captured the relevant artefacts and respective judges have judged them all, you have everything you need to move on to the next step, 4. Improve.