Ensuring a Trustworthy Evaluation Set

With an initial set in place, take a step back and consider whether you can trust the data to support the conclusions you want to draw.

We introduce three checks to help you:

Check 1RepresentativenessMeaningful coverageDoes the set meaningfully cover the users, topics, tasks, and variations in the evaluation design?Check 2Realism and accuracyQuality aligned with human judgementDo the cases make sense, and is their reference information correct?Check 3Cost-effectivenessLow redundancy and focused cases Does each case provide useful evidence about the behaviour being measured?

1. Representativeness: Meaningful coverage

Your evaluation set can be viewed as a sample that proxies for real use, so it should meaningfully cover the range of users, topics, requirements and conditions you care about. In doing so, results from your evaluation will reflect more than just the cases you happen to test.

Checking representativeness means looking at your set through two lenses that work together:

• Distribution and sampling determine which scenarios go into the set and how much each is represented, so the overall mix aligns with what you set out to measure.
• Variety checks whether there is sufficient diversity within those scenarios, so results reflect the different forms each can take rather than resting on a single, more uninformative version.

Distribution and sampling

The distribution of test cases determines how much representation different user groups, scenarios, topics, and conditions receive in your evaluation set, and should be informed by your evaluation objectives and the specifications identified earlier.

To assess typical performance, you may want the set to reflect the users and tasks your application is expected to serve. Alternatively, you may deliberately give greater weight to certain scenarios based on their importance or risk, such as adversarial prompts, sensitive topics, should the product context be a high-stakes one.

Once you have determined the intended distribution, sampling is how you select cases from your available data to achieve that mix. Here are some approaches you can use:

Targeted selection

Choose cases with known failures or risks by hand. For example, pick:

• Past failures: cases the system got wrong before.
• Reported risks: inputs users or reviewers have flagged.
• High-stakes requests: where a wrong answer is costly.

Use when: You investigate a specific weakness.

Stratified sampling

Split the data into groups, then pick cases at random from each group. For example, group by:

• Demographic group: match your user base.
• Time period: sample from each week or month.
• Label or outcome: balance successful and failed requests.

Use when: Each group must be in the set, in equal numbers or in proportion to your users.

Simple random sampling

Every case has the same chance of being picked. For example, sample from:

• Production logs: a slice of recent real traffic.
• A large unlabelled pool: before investing in labels.
• A new feature: an unbiased first look at its usage.

Use when: You need a small sample from a large amount of data. A small sample can miss rare cases.

Check your distribution

After sampling, check whether your evaluation set reflects the intended distribution across the scenarios and groups you identified.

If important areas are missing or underrepresented, revisit Sourcing Evaluation Data to address those gaps.

Variety

Distribution fixes the mix of scenarios; variety works within each one. A single scenario can reach your system in many forms, and a trustworthy result should hold across them rather than depend on one tidy version. So within the categories distribution has already chosen, diversify your cases to test robustness in two ways: vary what the input contains, and vary where it comes from and the intent behind it.

Vary your test cases

Input contentWhat the input contains

• Style & LanguageHow the same request is worded.Formal, casual, or Singlish; typos, misspellings, or alternative keywords.
• Structure & FormatHow the request is laid out.A single paragraph, a list of fields, or several messages.
• Detail & CompletenessHow much the request gives you to work with.Missing details, ambiguous asks, or far more information than needed.

Input source and intentWhere it comes from, and the underlying aim

• Adversarial intentInput that sets out to make the system misbehave.Jailbreak-style phrasing, or hidden instructions in a pasted document.
• Non-user sourceInput that did not come straight from the end user.An uploaded document, retrieved passages, or an upstream system.
• Shifting goalsA request whose aim moves as the interaction unfolds.Needs that evolve across turns, or mixed with unrelated chatter.

2. Realism and accuracy: Quality aligned with human judgement

Next, examine the individual test cases in further detail to check for data quality: Are they natural, realistic, and appropriate for the task?

Even with good coverage, poorly designed test cases can undermine the reliability of your evaluation results.

To help you assess data quality, we suggest the following steps as a practical starting point, rather than an exhaustive checklist:

Revisit your diversity parameters.

Keep variation realistic for the setting you are evaluating.

• Reflect real user behaviour, such as short, casual messages with typos.
• Adjust the input style, structure, or amount of detail.
• Review the revised cases so the set stays varied and realistic.

Use an LLM judge for a first pass, if possible.

Give the judge enough context, then review what it flags.

• Provide the application context, evaluation objectives, and example test cases.
• Ask it to flag unnatural phrasing, inconsistent details, ambiguous requests, or questionable assumptions.
• Treat its findings as signals for review, not definitive judgements.

Then review with people.

Bring in people who know the domain.

• Have domain experts, or people familiar with the application, review the cases.
• Prioritise flagged or high-risk examples, and spot-check some unflagged ones.
• Verify scenarios are appropriate, details accurate, and reference answers backed by reliable sources.

Worked example: The same conversation, in different styles

Can you spot the difference?

Too clean

Typical of generated cases

I am self-employed and applying for the housing grant. What documents do I need to submit?

You will need proof of identity, proof of address, and your latest tax assessment.

Thank you, that is clear.

The user supplies every detail at once, in the order the system needs it, and gets the answer on the first try.

How it actually goes

Typical of observed cases

what docs do i submit

Could you tell me which application you are asking about?

housing

You will need proof of identity, proof of address, and your latest income documents.

i dont have payslip

If you are self-employed, please provide your latest tax assessment instead.

where to upload

The user starts with one word, answers half the question, and changes direction at the end.

While both conversations are about the same topic and reach the same answer, the second one typically matches the style and flow of real users, which makes the evaluation more trustworthy.

3. Cost-effectiveness: Low redundancy and focused cases

Finally, keep your evaluation set lean and meaningful to improve evaluation efficiency. Every test case has a cost, whether in compute, execution time, or human review. A larger set is only better when the extra cases provide new evidence about the system's behaviour.

A few practices help:

Remove duplicates

Run deduplication checks to find exact copies and prompts with similar meaning. Review similar cases before removing them: a change in wording or context may test a different requirement. Keep the variations that add useful evidence.

Consider removing very simple cases

Cases that are too simple tend to tell you little about the behaviour you care about. Keep them only when basic functionality, or the mix of everyday requests, is part of your evaluation objective.

Keep complexity purposeful

Make sure each case's complexity serves what you intend to measure. Avoid unnecessary information, irrelevant tasks, or excessive input and output that consume resources without contributing to the judgement. For example, testing information extraction may need only a short document with the relevant fields, whereas testing reasoning across sources may justify more extensive inputs.

An efficient evaluation set balances coverage with resource use, directing more effort towards the cases that provide additional or more valuable evidence.

---

Iterating on the evaluation set

In practice, you may need several passes to get a dataset that is representative, realistic, and efficient.

Before you head off

Check that you have the following from this sub-chapter:

• A trustworthy evaluation set that is representative, realistic, and efficient.
• Understanding where and how to source your evaluation set.

If you are ready, move on to the next chapter, 3. Execute, to learn how to execute your system and evaluation in a reliable and consistent manner.