Artefacts for Judging and Possible Judgements
Before choosing a judge, decide what it will judge and what it should return.
This page covers both: the artefact that goes in, and the judgement that comes out.
The artefact for judging
The artefact is everything you hand to the judge. It has two parts:
- Subject of judgement: The piece of system behaviour or output being judged
- Reference (optional): any extra information the judge needs to judge that subject fairly
Subject of judgement
The subject is the specific piece of your system's behaviour you put in front of the judge. Almost anything your system produces can be a subject, and there is no fixed set of dimensions for describing one.
Two questions are useful starting points for deciding what to put in front of the judge:
- Target: What output/behaviour of the system are you judging?
- Scope: How much of that output/behaviour are you judging together?
Picture a subject as a selection on some part of your system: the target is where you look, and the scope is how much you take in.
To illustrate this further, consider the diagram below, where each row is one run of your system, and each numbered bracket marks one possible subject.
- A single stepOne intermediate step, judged on its own.
- A span of stepsSeveral consecutive steps, judged as a sequence.
- The outputWhat the system returns to the user.
- The whole runEverything from input to output, judged end to end.
- Several runsMultiple separate runs judged together, such as one user's sessions over time.
Across alternatives: you can also put runs side by side instead of in sequence, such as two candidate outputs for the same input, or repeats of one input.
To judge an internal trace, you can run the whole system and capture that step, or run the component on its own. This depends on the granularity in which your test cases are defined, which is covered later under 2b. Evaluation Set Design.
Reference
Sometimes the subject alone is not enough for a fair judgement. The reference is the extra information the judge needs to complete it.
It typically comes in three kinds, each answering a distinct question:
| Kind | Question it answers | Examples |
|---|---|---|
| Task context | What was the system supposed to do? | User requestSystem prompt / instructionsTask specificationApplicable policy |
| Contextual evidence | What information do I need to understand or verify what it did? | Source documentRetrieved passagesTool outputsPrior conversation turns |
| Expected output | What should the answer or action have been? | Gold answerReference labelExpected tool callTarget trajectory |
What reference an artefact needs, if any, depends on the judge you choose; the next chapter covers the options.
Possible judgements
A judgement is what the judge returns: a label for the artefact, in a form that is measurable and ready to be aggregated and interpreted later.
Each judgement takes one of a few broad shapes:
- NominalDistinct buckets with no orderPass/FailPositive/Negative
- OrdinalBuckets with meaningful orderPoor→excellentLow/Medium/High Risk
- DiscreteDistinct, countable valuesNo. of errors1-5 Likert
- ContinuousReal valuesConfidence Score [0, 1]Probability [0, 1]
- OrderingOutputs placed in relative orderA > B1st → 2nd → 3rd
- Scaled comparisonDifference expressed by degreeA better than B by 20%B worse than A by 1 pt
- StructuredText in predefined fieldsIssue + Evidence + FixStrengths + Weaknesses
- Free-formOpen-ended proseReviewer commentsWritten critique
Putting it together
To summarise: an artefact goes in, and a judgement comes out. The judge is what maps one to the other. Here are some examples of how the pieces combine:
An artefact can bundle several outputs, such as a whole conversation or a set of candidates to rank, but each act of judging still takes one artefact in and returns one judgement.
Was this page helpful?