Evaluation methods

After defining your evaluation criteria and scoring scale in your evaluation plan, you may set up your evaluation method. The right method depends on what you're trying to test and what level of automation you need.

Choose an evaluation method

More rigidMore automationMore flexibleLess automation

1. 1Deterministic checksExact match, regular expressions (regex)
2. 2Embedding similaritySemantic similarity to a reference
3. 3LLM-as-a-judgePolicy-based model scoring
4. 4Human evaluationContextual expert judgement

The methods are not mutually exclusive. For example, you might check an output with deterministic rules or an LLM-as-a-Judge before escalating ambiguous cases for human review.

Method	Best suited to	Advantages	Limitations
Deterministic checks	Outputs with a known format or set of permitted values	Fast, inexpensive, repeatable, and easy to audit	Brittle for open-ended responses; checks do not account for meaning and context
Embedding similarity	Tasks where a correct answer should be semantically close to a reference	Scales well and recognises paraphrases that exact match would reject	Similarity is not correctness or completeness and can only be applied to certain criteria
LLM-as-a-judge	Open-ended outputs that can be assessed against a rubric	Flexible, scalable, and able to consider several aspects of an answer	Subject to LLM biases and inconsistencies
Human evaluation	Contextual, subjective, or high-risk judgements	Most reliable and captures nuances well	Slow and costly; human fatigue; reviewers can disagree or apply the rubric inconsistently

Deterministic checks

• Exact match compares the output with an expected answer. Normalise permitted differences such as whitespace or letter case before comparing.
• Regular expressions check for defined formats, required phrases, or prohibited strings.

Example: Apply deterministic checks

Reference format

GrantsAssist requires application references to use GRANT- followed by six digits. The team checks each reference with:

^GRANT-\d{6}$

This accepts GRANT-482193 and rejects values such as 482193, GRANT-48219, or GRANT-ABC123.

Response structure

GrantsAssist must return an eligibility summary in this structure:

## Eligibility
Based on the information provided, ...

## Next steps
To continue, ...

assert headings == ["## Eligibility", "## Next steps"]
assert paragraphs[0].startswith("Based on the information provided,")
assert paragraphs[1].startswith("To continue,")

Embedding similarity

Compare the generated output with a reference using an embedding model. Treat the result as semantic similarity, not proof of correctness. A similar answer can still contain incorrect details.

Example: Compare an answer with a reference

import numpy as np

reference = "Applicants must be at least 65 years old."
response = "Applicants aged 65 and above qualify."

# Apply your preferred embedding model
reference_vector = embed(reference)
response_vector = embed(response)

similarity = np.dot(reference_vector, response_vector) / (
    np.linalg.norm(reference_vector) * np.linalg.norm(response_vector)
)

print(f"Cosine similarity: {similarity:.3f}")
# Example output: Cosine similarity: 0.820
# Set the similarity threshold using annotated examples.

LLM-as-a-judge

An LLM-as-a-judge uses a language model to score another model's output. It can scale open-ended evaluation beyond what deterministic checks and embedding similarity can capture, but its scores are useful only after the judge has been well validated.

Set up an LLM judge

1. Use an independent judge where possible.

   • To reduce self-preference bias, select a model from a different model family from the one that generated the response.

2. Evaluate one criterion at a time.

   • Asking one judge call to score several criteria can reduce performance.
   • Ask the judge to return a structured response based on your policy.

3. Provide evidence and representative examples.

   • Give the judge the full context about the application, its target users, and all its inputs.
   • Include reference few-shot examples.

4. Set the temperature to 0 where possible.

   • Although a temperature of 0 does not guarantee identical results, judge calls should be as consistent as possible.

Other tips

• Run the same examples more than once to ensure that the judge is consistent.
• Run multiple judges on the same task to reduce dependence on one model's biases.
• You may include a confidence score in the judge output, but treat self-reported confidence with caution because it may not be reliable.
• Consider pairwise comparison when choosing between two candidate outputs instead of asking for a zero-shot rating.

Example: Score answer accuracy

import json

JUDGE_MODEL = "your-pinned-judge-model-version"

# Example labels: accurate | partially_accurate | not_accurate
JUDGE_PROMPT = """
## Instructions
Evaluate the candidate response.

<policy> {evaluation_policy} </policy>

## Reference
<context> {application_context} </context>
<input> {inputs} </input>

## Response
<response> {response_to_evaluate} </response>

## Evaluation criteria
<criteria> {evaluation_criteria} </criteria>

Return JSON only:
{{
    "score": "<label defined in evaluation_criteria>",
    "reason": "<brief reason>"
}}
"""

def judge(...):
    judge_prompt = JUDGE_PROMPT.format(...)
    result = client.messages.create(
        model=JUDGE_MODEL,
        temperature=0,
        max_tokens=300,
        messages=[{"role": "user", "content": judge_prompt}],
    )
    return json.loads(result.content[0].text)

What to watch out for

LLM judges can be influenced by factors unrelated to the evaluation criteria. Understanding these limitations can help you configure your judge more carefully.

• Self-preference bias: The judge favours outputs produced by itself or a related model.
• Inherited bias: Biases from training data affect judgements across demographic, cultural, or language groups.
• Position bias: The judge favours an answer because it appears first or second.
• Style bias: The judge rewards a familiar tone, format, or writing style over substance.
• Verbosity bias: The judge rewards a longer answer even when it is no more correct.
• Authority bias: Confident, concrete, or authoritative-sounding details receive undue weight.
• Inconsistency: Repeated calls produce different scores for the same example.
• Overconfidence: The judge reports high confidence for an incorrect or weakly supported score.
• Sycophancy: The judge mirrors a stated preference instead of applying the criteria independently.

For an overview of known biases and reliability challenges, see A Survey on LLM-as-a-Judge.

Judge calibration (measurement)

This step requires human review on a subset of examples.

How many examples should you review?

At the very least, we recommend preparing a human-reviewed dataset of 30–50 examples. Based on research and practical experience, this provides a reasonable balance between annotator fatigue and sample size. Use the annotated set to calculate agreement between the reviewers and the judge.

What is annotator fatigue?

Human review can suffer from fatigue-related inconsistencies. While larger review sets can produce more robust reliability estimates, reviewing large sets frequently can become impractical and may lead reviewers to skip calibration altogether.

This is not a hard rule.

The amount of data needed depends on the use case and scoring rubric — for instance, subjective tasks or multi-class labels may require more examples for calibration, and subjective tasks may require more annotators. More advanced teams should run statistical tests to estimate the sample size needed for their use case.

With a small sample, a few disagreements or rare cases can substantially change agreement scores. Prepare more annotated examples where possible, provided they are meaningful and representative of the target population.

Classification metrics

Measure categorical judgements with traditional classification metrics, depending on your use case.

True positive rate (TPR) and true negative rate (TNR)

TPR, also known as Recall, and TNR measure how often the judge correctly identifies passing and failing responses, respectively. Reporting them separately can reveal a judge that favours one label.

Accuracy

The overall proportion of human labels the judge matches.

Precision

Useful for important classes, especially when some labels are uncommon or some errors are more serious than others.

F1 score

Combines precision and recall into one score using their harmonic mean. It is useful when both missed positives and false alarms matter, especially when classes are imbalanced.

Inter-annotator agreement metrics

Measure how consistently human and LLM annotators agree on the same labels.

Cohen's kappa

Measures agreement between two annotators on categorical labels while accounting for agreement that could occur by chance. This gives a more informative result than raw percentage agreement when one label is especially common.

Krippendorff's alpha

Like Cohen's kappa, Krippendorff's alpha accounts for chance by comparing observed disagreement with the disagreement expected by chance. Use it instead of Cohen's kappa when you have more than two annotators, some annotations are missing, or your scores are ordinal or continuous rather than categorical.

Alternative Annotator Test (Alt-Test)

The Alt-Test is one way to measure the alignment of your LLM-as-a-Judge. It reframes the goal of evaluation from "Is the model correct?" to "To what extent do LLMs concur with human annotations?"

Essentially, it is a leave-one-annotator-out hypothesis test that measures whether an LLM judge agrees with the remaining human consensus at least as well as the left-out human does.

How is this better than traditional metrics? Commonly used agreement measures (e.g. Cohen's kappa, Krippendorff) only assess agreement among annotators, and performance metrics (e.g. Accuracy, F1 Score) only evaluate whether the LLM matches human performance. The Alt-Test provides two key advantages:

• It is actionable: a high winning rate provides statistical evidence that the model can stand in for human annotators, and the advantage probability provides a measure to compare between models.
• It captures the variability amongst humans themselves, accounting for the fact that humans disagree with each other.

Apply these metrics

See the hands-on implementation and extension, the original Alt-Test paper, and GovTech's MetaEvaluator for tools and further detail.

Judge calibration (optimisation)

Using the set of human-reviewed examples, calibrate the judge to be aligned to your annotations. Re-run the alignment metrics after each material change to measure judge improvements.

Judge optimisation approaches

The approaches below include prompt optimisers from the DSPy library.

Approach	How it improves the judge	Data guidance
Few-shot examples	Demonstrate how human reviewers apply the criteria	3-5 examples covering key labels
COPRO	Refines judge instructions	No fixed minimum; use a representative optimisation set
MIPROv2	Optimises instructions and few-shot examples together	No fixed minimum; the default configuration requires at least 35 validation examples
GEPA	Uses execution feedback to improve prompts	No fixed minimum; use the smallest representative validation set and keep the training set as large as practical
MemAlign	Learns principles and examples from human feedback	Start with at least 10 examples containing human feedback, with a mix of positive and negative labels

Avoid optimising on your test set

Use separate sets for prompt optimisation and final validation. Reporting performance on the same examples used to optimise the judge will overstate how well it generalises.

Human evaluation

Human evaluation is appropriate when outputs are open-ended, contextual, or difficult to score reliably through automated methods.

When to use human evaluation

Human review is especially important:

• In high-risk use cases where the consequences of an incorrect judgement are serious.
• When the task requires domain expertise or specific contextual judgement.
• When the assessment is subjective.

Before starting the human review...

• Align annotators on the product use case, evaluation criteria, policy and taxonomy. Give them the full context, including the same information that an LLM judge would receive.
• Set a manageable review load and allow breaks to reduce fatigue.
• When using multiple annotators, define how disagreements will be adjudicated, but retain and examine them: disagreement may reveal unclear criteria, gaps in reviewer alignment, or cases that require specialist judgement.