Judgement Design
This sub-chapter covers the first half of your evaluation design: how to measure each specification from that list.
From judgements to metrics
A metric summarises many individual judgements. Measuring a specification therefore starts with a single judgement run:
- Take an artefact: a piece of your system's work, such as a tool call it made or a final response it gave.
- Put the artefact in front of a judge: a rule, a model, or a person applying your criteria.
- Record the judgement that comes out: a label such as
pass, a score, or a written critique.
A single judgement only tells you how the system performed on one test case. To assess performance more broadly, you need to evaluate it across multiple test cases, collectively known as an evaluation set.
Aggregating the resulting judgements produces metrics that summarise system performance. These metrics can then be interpreted against your acceptance criteria, which translate your definition of "good" into measurable standards, to give a definite answer on how well your system performed.
How this chapter is organised
We follow the framework from the inside out. First, the two ends of a judgement run: the artefacts you can put in front of a judge, and the forms a judgement can take. Then the types of judges that connect them. Finally, how judgements combine into metrics and how to interpret and communicate the results.
- Artefacts for judging & possible judgementsWhat goes in front of the judge, and what comes back?
- Types of judgesWho, or what, makes the judgement?
- Metrics & interpretationHow do judgements become a metric, and what does it tell us?
What you will have by the end
A judgement design for each specification on your list, made up of:
- Artefacts and possible judgementsWhat goes in front of the judge, and the form the judgement it returns takes
- A judge for each specificationThe type of judge (rule-based, model-based or human) that makes each judgement
- Metrics with acceptance criteriaHow the judgements combine into metrics, and the thresholds that decide whether your system is "good" enough
Was this page helpful?