Understanding & Sourcing Evaluation Data
With your data requirements established, the next step is to source the data needed to build your evaluation set.
When sourcing data, note that you are primarily gathering system inputs, sometimes accompanied by references, such as expected answers or source documents.
Explore the cards below to understand the different sources of evaluation data, and which may be most suitable for your evaluation set. The three are not mutually exclusive, so combine them where it helps build a more comprehensive set.
Working with benchmark data
Benchmarks give you a starting point with defined tasks and scoring rules. Some cover broad model capabilities; others focus on a specific domain, language, or workflow. Internal benchmarks can capture requirements specific to your organisation.
| Coverage | Examples |
|---|---|
| General capabilities | MMLU-Pro, GPQA Diamond, Humanity’s Last Exam (HLE), FrontierMath, MMMU |
| General agentic task completion | GAIA, Terminal-Bench, AgentBench |
| Software engineering | SWE-bench Verified / SWE-bench Pro |
| Tool use / function calling | τ-bench, BFCL |
| Domain-specific tasks | Medical, legal, financial, cybersecurity, or other specialised evaluation datasets |
| Localised content | RabakBench for localised multilingual toxicity; SGHateCheck for hate speech in Singapore |
Start by checking which tasks match your evaluation design. You can adapt cases with local names, situations, or phrasing, then review whether the expected response has changed. Check for outdated information. Use the trust checks to assess the cases you select.
Keep the meaning of the case the same. When you adapt cases with local names, situations, or phrasing, check that the change does not alter what the case tests. If the meaning changes, the expected response may no longer be correct. Record every adaptation you make.
The model may have seen the benchmark already. Public benchmarks often appear in training data. A high score can show recall instead of capability. Check for overlap with training data before you trust the result.
Using curated or historical data
Relevant historical interactions are often a strong starting point for evaluating actual use. Support conversations, user feedback, and application logs can show how people express their needs and where the system has failed. You can also write cases with people who know the application to cover local requirements and known risks.
- Select relevant records. Choose data from workflows and system versions that fit your evaluation objective. Apply the required privacy and access controls.
- Sample for useful coverage. Group records by task, topic, language, or outcome, then sample within these groups. Include known failures and less frequent cases when the objective calls for them.
- Validate the cases. Check that the context is still relevant and that facts and reference outputs remain correct. A recorded system response is not automatically a correct reference answer.
- Keep the set current. Review affected cases when policies, source information, or application behaviour change.
For example, a past eligibility question may still be useful after a scheme changes. Its reference answer needs to match the scheme rules used in your current evaluation.
Use the trust checks to review the sample’s coverage, realism, and cost-effectiveness. Historical data can miss new features or rare risks, so add other sources where needed.
Creating useful synthetic data
Synthetic data helps you build cases for the gaps in the data you already have. You can adapt existing cases or generate new ones from a clear specification. The value comes from the cases you validate and keep.
- Define the target. Give the generator the task, required input format, relevant context, and behaviour you want to test.
- Provide seed examples. Use reviewed examples that show the language, detail, and range of situations you need.
- Vary specific parameters. Change input style, context, or difficulty to fill identified gaps. Check whether each change also changes the expected response.
- Review a small batch first. Refine the instructions and seed examples before generating more cases. Check facts, reference answers, and realistic combinations of details.
- Filter and review again. Remove duplicates and cases that add little evidence. Keep a record of how the cases were generated.
For a booking agent, you might start with a complete booking request, then generate variations with a missing date, conflicting time slots, or Singlish phrasing. Each variation should test a clear requirement.
Apply the representativeness, realism, and cost-effectiveness checks before adding generated cases to your evaluation set.
For tools, see Kaleidoscope for persona-based test generation and KnowOrNot for cases that test questions outside the system's knowledge base. The Responsible AI Playbook contains both guides.
Consider existing benchmarks and curated or historical data first. If they do not meet your requirements or leave important gaps, synthetic generation can help supplement your evaluation set.
See Building your testing dataset in the Responsible AI Playbook to understand more. This safety test set combines existing benchmarks, manually created cases for known risks in the Singapore context, and synthetic creation and curation, including AI-generated jailbreak variations.
Was this page helpful?