Skip to main content

Evaluation Set Design

Recap: what you bring into this chapter

You should have completed the first half of your evaluation design: the Judgement Design.

If you don't have that yet, complete that chapter first.

This sub-chapter covers the second half of your evaluation design: the set of test cases you run the measurement over. The subjects you judge, together with the artefacts your judge reads, are collectively the evaluation set.

Building a trustworthy evaluation set​

An evaluation set is only useful if it has construct validity: how well your results support meaningful conclusions about the real system behaviour that matters. This means testing relevant scenarios and providing the evidence needed to assess the specifications that you defined earlier.

We came up with a three-step process to help you do so:

Building a trustworthy evaluation setThree stages run left to right: define the data requirements, build the set by sourcing and curating test cases, and validate trustworthiness against representativeness, realism and accuracy, and cost-effectiveness. When the criteria are met the set is ready for evaluation. If a check fails, loop back over the top: a requirement or coverage gap returns to the requirements stage, and unsuitable or inaccurate cases return to the build stage.Requirement or coverage gapUnsuitable or inaccurate cases01Define requirementsSystem inputs & outputsArtefact to judgeExpected judgement02Build the evaluation setSource, create and curatethe test cases.03Validate trustworthinessRepresentativenessRealism and accuracyCost-effectivenessAll 3 checks passed!Ready for evaluation!
Repeat until you trust the set

Building a trustworthy evaluation set is iterative; if any check fails, return to the step that fixes it and repeat the checks until you pass all three.

How this chapter is organised​

We follow the three-step process in order, each with its own page, iterating until your evaluation set is ready to use:

  1. Data requirementsWhat must each test case contain?
  2. Sourcing evaluation dataWhere do the test cases come from?
  3. A trustworthy evaluation setIs the set representative, realistic, and effective?

What you will have by the end​

An evaluation set that completes your evaluation design, supported by:

  • A defined data schemaWhat each test case contains: the system input and output, the artefact your judge receives, and the judgement it returns
  • A sourced evaluation setTest cases drawn from benchmarks, synthetic generation, or curated and historical data
  • Evidence the set is trustworthyThe set passes the representativeness, realism and accuracy, and cost-effectiveness checks
Last updated on

Was this page helpful?