Types of Judges
After understanding what goes in and out of a judge, we can start to understand the connector, the judge: it applies your criteria to an artefact and returns a judgement.
While not exhaustive, most judges fall into three broad families. They sit on a spectrum, from fully deterministic checks to flexible human judgement:
What's needed
- Ground-truth reference or formal specification
- Parser to extract relevant outputs
When it's used
- You need a simple, highly repeatable check.
- Time or compute for evaluation is limited.
Strengths & limitations
- Cheap in time and compute
- Perfectly reproducible
- Transparent: easy to see how a verdict was formed
- Limited on nuance and open-ended quality
- Rigid: requires manual maintenance
Best practices
- Normalise before you compareTrim whitespace, fix case, and standardise formats so trivial differences don't fail a correct answer.
- Prefer tolerances over exact match for numbersCompare in a bounded manner rather than demanding identical outputs.
- Review failures for false negativesSample what the rules reject; false positives slipping through calls for a revision in the rule set.
What's needed
- A scoring model
- Written rubric (optional)
When it's used
- Outputs are open-ended but a model can still judge them consistently.
- Evaluation must scale beyond human review.
Strengths & limitations
- Automates evaluation workflows
- More nuanced than rule-based checks
- Potentially biased
- Non-deterministic unless pinned
Best practices
- Match the model judge as closely to the judgement task as possibleSpell out each score level with a short description and, ideally, an anchor example.
- Validate against human labelsModel judges serve as a proxy for human judgement. A judge you have not calibrated against human review is an assumption, not a measurement; see the testing judges guide below.
What's needed
- The right assessors
- Clear annotation guidelines
When it's used
- High-risk cases where a wrong judgement is costly.
- Judging highly open-ended quality.
- Novel tasks with no reference or trained evaluator yet.
Strengths & limitations
- Captures subtlety no automation can
- The only ground truth for real users
- Slow and expensive
- Hard to scale
- Annotator variance to manage
Best practices
- Write clear annotation guidelinesUse a rubric and worked examples; run a pilot before the full round.
- Use multiple annotators per itemDefine how disagreements will be adjudicated, but retain and examine them; measure how far reviewers agree with a chance-corrected metric (see the testing judges guide below).
- Set manageable review load and allow breaks to reduce fatigue.Fatigue builds over a long session and quietly pulls ratings off the rubric.
- Align annotators on product use case, evaluation criteria, policy & taxonomy.Give annotators the full context, including the same information that an LLM judge would receive.
One judge that has grown rapidly in popularity is LLM-as-a-judge. While flexible, poor understanding or setup can significantly reduce judgement quality. The primer below explains the key concepts and practices.
Note: A Primer on LLM-as-a-judge
A model-based judge that uses a large language model (LLM) to apply your custom rubric to an artefact.
You give the LLM a single prompt, for example:
Wrap each part of the prompt (role, task, rubric, artefact, response format and examples) in a clear delimiter, such as XML tags (<rubric>…</rubric>, <answer>…</answer>). This is surprisingly important. Without clear boundaries, the lines between parts can blur, and the judge may confuse one part with another. For example, an answer can contain its own headings, lists or JSON, so the judge cannot tell where the answer ends and the next part begins.
LLM judges come with inherent biases unrelated to the evaluation task. These biases can be reduced but may never be removed completely, so watch for them:
- Self-preference bias: The judge favours outputs produced by itself or a related model.
- Inherited bias: Biases from training data affect judgements across demographic, cultural, or language groups.
- Position bias: The judge favours an answer because it appears first or second.
- Style bias: The judge rewards a familiar tone, format, or writing style over substance.
- Verbosity bias: The judge rewards a longer answer even when it is no more correct.
- Authority bias: Confident, concrete, or authoritative-sounding details receive undue weight.
- Inconsistency: Repeated calls produce different scores for the same example.
- Overconfidence: The judge reports high confidence for an incorrect or weakly supported score.
- Sycophancy: The judge mirrors a stated preference instead of applying the criteria independently.
For an overview of known biases and reliability challenges, see A Survey on LLM-as-a-Judge.
-
Use an independent judge where possible.
To reduce bias, use a model from a different family than the one that produced the artefact. -
Evaluate one criterion at a time.
Asking multiple criteria at once can lead to confusion and inconsistent results. -
Provide evidence and representative examples.
Give the judge full context on the system, evaluation criteria, and all inputs. Start with a few representative examples through few-shot prompting (e.g. 3-5), then validate and adjust based on judge performance. -
Calibrate the judge against human labels.
Ensure that LLM judges agree with trusted human judgements. See the testing judges guide below for more.
LLM judges can be calibrated to align to any human annotated dataset you have. Explore these approaches, including prompt optimisers from the DSPy library, and re-run agreement metrics from the testing judges guide to measure judge improvements:
| Approach | What it does | Data to start |
|---|---|---|
| COPRO | Refines the judge's instructions | A representative labelled set |
| MIPROv2 | Optimises instructions and few-shot examples together | At least 35 validation examples (default) |
| GEPA | Improves the prompt from execution feedback | A large training set |
| MemAlign | Learns principles and examples from human feedback | At least 10 examples, mixing positive and negative |
Use separate sets for optimisation and final validation; reporting on the examples you optimised on overstates how well the judge generalises.
How do we know if we can trust our judges? See below, especially when using model-based and human judges, where judgement might not be completely transparent.
Note: A Guide to Testing judges
Before relying on a judge, check that it measures the standard you intend. The appropriate checks depend on the type(s) of judges used:
- Model-based judge: Validate against trusted, human-labelled examples. (covered below)
- Agreement between judges: Check consistency between a model and trusted human labels, or among multiple human or model judges. (covered below)
- A single trusted expert: May establish the reference standard; document their criteria and decisions.
- Deterministic rule-based check: Validate that the rules and implementation correctly check the intended condition.
We recommend validating model-based judges against a trusted, hand-labelled subset of examples, starting with 30-50 of them, a reasonable balance between annotator fatigue and sample size based on research & practical experience.
How many you need depends on your use case and rubric. Subjective or multi-class tasks usually need more examples, and subjective tasks also need more annotators. To size the set precisely, run a power analysis for your use case.
With a small sample, a few disagreements or rare cases can swing the agreement score, so add more examples where you can, as long as they stay meaningful and representative of your target population.
To measure the judge's performance relative to the golden human labels in categorical judgements (a failure is a positive case!), here are a few classification metrics to consider:
| Metric | What it measures | When to use |
|---|---|---|
| Recall / True Positive Rate (TPR) (recommended) | Of the failures humans found, the share the judge also catches | Missing a failure is costly |
| True Negative Rate (TNR) (recommended) | Of the good outputs humans passed, the share the judge passes | False alarms are costly |
| Precision | Of the cases the judge flags, the share humans agree are failures | You act on each flag, or false alarms are expensive |
| F1 Score | Precision and recall combined into a single score | Both misses and false alarms matter, especially under class imbalance |
| Accuracy | The share of all cases where judge and human agree | For an overall match rate; add class-specific metrics when labels are imbalanced |
While classification metrics reveal the judge's errors against trusted categorical labels, they assume that the labels are golden. However, this assumption may not always be true.
We hence can measure how consistently the different judges align on the same labels. Below are some ways we recommend:
| Method | What it measures | When to use |
|---|---|---|
| Cohen's kappa | Chance-corrected agreement between two judges on categorical labels | Two sets of categorical judgements; use weighted kappa for ordinal ratings |
| Krippendorff's alpha | Chance-corrected agreement supporting multiple judges, missing annotations, and different scales | More than two judges, missing labels, or nominal/ordinal/numerical judgements |
| Alt-Test | Whether a model judge agrees with the human consensus as well as a held-out human does | Validating a judge against several annotators, crediting it when the humans themselves disagree |
See the hands-on implementation and extension, the original Alt-Test paper, and GovTech's MetaEvaluator for tools and further detail.
How each judge works in practice
To understand how each type of judge works in practice, we can look at some worked examples.
These examples below use GrantsAssist, a system that answers grant eligibility questions, to show how these designs come together in practice. Each example starts with a summary of its design: the specification to measure, the metric already chosen to express it, and then the artefact, the judgement, and the judge they imply.
The choice of judge comes after the choice of metric, not before.
The specification you intend to address should inform the metric of choice. The metric then fixes the judgement each case must return, and the judge is whatever can produce that judgement.
If you pick the judge first, you might end up measuring what is convenient and not what matters.
Worked example (rule-based): Validating structured grant responses
Judgement design
GrantsAssist looks up a grant application and returns an eligibility summary with a cited grant reference.
SubjectReference + summary (3 · the output)
ReferenceNone needed
Rule-based
The reference and the structure each have exactly one correct form, so the criteria are explicit and unambiguous, and the checks run cheaply at high volume.
Validate application references
- References must be
GRANT-followed by six digits:
^GRANT-\d{6}$
- Accepts
GRANT-482193; rejects482193,GRANT-48219,GRANT-ABC123. - Pasted references carry trailing whitespace; strip it before matching, or a valid reference fails on formatting alone:
reference = reference.strip()
assert re.fullmatch(r"GRANT-\d{6}", reference)
- The reference is an identifier, so exact match fits. A funding amount would compare within a tolerance instead:
assert abs(parsed_amount - expected_amount) <= 0.01
Validate response structure
- The summary must follow a fixed structure:
## Eligibility
Based on the information provided, ...
## Next steps
To continue, ...
- Parse and check it programmatically:
headings = [heading.strip() for heading in headings]
paragraphs = [paragraph.strip() for paragraph in paragraphs]
assert headings == ["## Eligibility", "## Next steps"]
assert paragraphs[0].startswith("Based on the information provided,")
assert paragraphs[1].startswith("To continue,")
- Sample what the check rejects; this helpful reply fails the assertion above on capitalisation alone:
## Eligibility
Based on the information provided, you are eligible.
## Next steps
To continue, submit form GA-2.
- That is a false negative in the rule, not the output; lower-case before comparing:
assert [h.lower() for h in headings] == ["## eligibility", "## next steps"]
Worked example (model-based): Scoring free-text answers by semantic meaning
Judgement design
GrantsAssist also answers free-text questions, e.g. "Can I apply as a sole trader?".
SubjectFree-text answer (3 · the output)
ReferenceApproved reference answer (expected output)
Model-based
There is no single correct wording, so exact-match rules cannot judge it; an embedding model turns closeness in meaning into a number, scored consistently and scaling beyond hand review.
Score by semantic similarity
- The label is the numerical representation of the answer's meaning.
- Embed a reference and the candidate answer, then take their cosine similarity:
from sentence_transformers import SentenceTransformer, util
model = SentenceTransformer("all-MiniLM-L6-v2")
reference = model.encode("Sole traders can apply if registered for GST.")
candidate = model.encode("Yes, if you're a sole trader registered for GST, you're eligible.")
score = util.cos_sim(reference, candidate).item() # e.g. 0.81
- The score only means something against one fixed reference per question; curate that answer rather than compare to a rough idea of "good".
Validate against human labels
- Sample 30 to 50 real answers across the questions and styles you see, hand-label each, then set the labels beside the scores:
| Candidate answer | Score | Human verdict |
|---|---|---|
| Yes, a sole trader registered for GST is eligible. | 0.81 | Correct |
| No, a sole trader registered for GST cannot apply. | 0.79 | Wrong |
| You'll need an active GST registration to qualify. | 0.66 | Correct |
- The middle row exposes the problem: a flatly wrong answer scores almost as high as a right one.
- The cause is the model itself; a general-purpose embedding model rates topical overlap, and this pair differs only by a negation, which barely shifts the vector. It answers "same topic?", not "correct answer?".
- The fix is a different model, not a tweak; an entailment (NLI) model reads the negation as a contradiction. Swap it in, then re-check against the same labels.
Watch for bias, and keep scores repeatable
- This model's pull toward lexical overlap is itself a bias; an LLM judge adds others such as length, position, and self-preference. Check for them on the same labelled set.
- If the judge is non-deterministic, pin it or average repeats so the same answer does not drift between runs.
Worked example (human): Rating how well a refusal is handled
Judgement design
Some GrantsAssist replies decline an application, which is high-risk: a cold or confusing refusal can leave a vulnerable applicant worse off.
SubjectRefusal reply (3 · the output)
ReferenceEligibility policy (task context)
Human
The judgement is high-risk and novel: there is no reference answer or trained evaluator for “handled a rejection well”, so a small panel reads each refusal and rates it.
Write clear annotation guidelines
- Give reviewers a rubric that names each level with a concrete example:
| Rating | Meaning | Example |
|---|---|---|
| 2: Good | Gives the reason, stays warm, offers a next step | "You're not eligible because your turnover exceeds the cap, but you may qualify for …" |
| 1: Adequate | Correct but curt, or missing a next step | "Your application does not meet the criteria." |
| 0: Poor | Inaccurate, confusing, or dismissive | "Request denied." |
- Pilot it on a few refusals first, and fix any wording reviewers read differently.
- Brief them on the product, eligibility policy, and taxonomy, so they rate against the bar the assistant faced rather than personal taste.
Use multiple annotators and measure agreement
- Put at least two reviewers on each refusal and measure how far they agree; Cohen's kappa is the default for categorical ratings like these:
| Refusal | Reviewer A | Reviewer B |
|---|---|---|
| #1 | Good | Good |
| #2 | Adequate | Poor |
| #3 | Good | Good |
- Low agreement points back to unclear guidelines, not bad reviewers.
- Cap each reviewer's load and allow breaks; fatigue quietly pulls ratings off the rubric over a long session.
Was this page helpful?