Defining Data Requirements

Previously, when building the Judgement Design, you worked from the artefact onwards to the judgement it returns, and the judge that connects them.

Here, you work further back up the chain to define the test cases that make up your evaluation set.

We can represent a test case as a single scenario you want to evaluate: you prepare a system input, and running it produces the subject of judgement, as part of the artefact to be assessed. Each case follows the same end-to-end flow outlined in the Overview of the Evaluation Design.

What are we evaluating?

The system in a test case can be whatever unit you are testing: a single component, a subsystem or the whole product, at the level of granularity you chose in Define. How to run that unit, and which version of it to run, is covered later in Execute.

Before building test cases, you need to establish what data is required and how it should be structured, particularly for these four elements:

Element	What to specify
System input	The data needed to run the system, including its fields and formats.
Subject of judgement	The system output or behaviour to capture for assessment, such as a final response or an intermediate trace.
Artefact	How the subject of judgement and any reference should be assembled for the judge.
Judgement	The structure of the judge's assessment, including its fields, formats and permissible values.

For each element, you will need to define the respective fields and formats. Together, these requirements establish what data to prepare, what to capture during execution, and how to structure the information for the judge.

---

To illustrate these concepts, we have provided three diverse examples below. Explore each to understand how data requirements differ across system types and evaluation objectives.

Worked example: Booking agent

• Product mission: Books appointments on behalf of users.
• Why evaluate: Improve the system: find where it picks the wrong tool or details.
• Specification: Calls the booking tool with the requested date, time and action.

2b · Evaluation set

System inputthe appointment request

{
  "requested_action":
  "Book appointment,
  12 Oct 2 p.m."
}

SystemBooking agent: selects a tool and calls it

Subject of judgementthe selected tool call, from the trace

{
  "trace_id": "…",
  "selected_tool_call": {
    "name":
      "book_appointment",
    "arguments": {
      "date": "2026-10-12",
      "time": "14:00"
    }
  }
}

assembled into the artefact+ the requested action

2a · Judgement design

Artefactrequested_action + the tool call

Judgerule-based check

find "book_appointment"
pass if
  date == 2026-10-12
  and time == 14:00

Judgementrule result

true

Schema

Field	Possible values
selected_tool_call.name	book_appointment, cancel_appointment, reschedule_appointment
arguments.date	any ISO-8601 date, YYYY-MM-DD
arguments.time	any 24-hour time, HH:MM
Judgement	true, false

Notes

• The subject of judgement comes from the agent’s trace, not a returned response: the booking itself is a side effect in the calendar system, so capture the whole tool call, its name as well as its arguments, rather than only the outcome.
• Store the expected date and time with each case; the rule needs them to decide pass or fail.

Collection requirement

This schema includes a trace. Connect the application to an observability or logging platform before you collect the evaluation data. This lets you capture the required intermediate outputs when each case runs.

Worked example: Chatbot

• Product mission: Answers questions about the documents an application needs.
• Why evaluate: Support a decision: is it ready to launch?
• Specification: Each reply directly answers the user's latest question.

2b · Evaluation set

System inputthe conversation so far

{
  "messages": [
    { "role": "user",
      "content":
        "When do you open?" }
  ]
}

SystemChatbot: replies to the user

Subject of judgementthe assistant reply

{
  "role": "assistant",
  "content":
    "We open at 9 a.m."
}

assembled into the artefact+ the conversation so far

2a · Judgement design

Artefactconversation + reply

JudgeLLM judge

Judgementverdict + reasoning

{
  "relevant": true,
  "reasoning": "…"
}

Schema

Field	Possible values
messages[].role	user, assistant, system
relevant	true, false
reasoning	<Free text>

Notes

• The reply only makes sense alongside the conversation, so the artefact carries the full messages array, not the reply on its own.
• An LLM judge is non-deterministic, so keep the reasoning field to audit and spot-check its verdicts.

Worked example: Classifier

• Product mission: Sorts each enquiry so it goes to the correct team.
• Why evaluate: Compare products: choose the classifier to deploy.
• Specification: Labels each enquiry accurately, as judged by a human.

2b · Evaluation set

System inputthe enquiry

"Where is my refund?"

SystemClassifier: assigns a label

Subject of judgementthe predicted label

"billing"

assembled into the artefact+ the reference label

2a · Judgement design

Artefactpredicted + reference label

Judgeexact match

pass if
  predicted == reference

Judgementcomparison result

"MATCH"

Schema

Field	Possible values
label (predicted and reference)	billing, shipping, account, technical, other
Judgement	"MATCH", "NO MATCH"

Notes

A classification case can be as small as one string in and one label out.

• You supply the reference label yourself: it is ground truth, not something the system produces.
• Normalise the label strings, such as case and whitespace, so a correct answer is not marked "NO MATCH" on formatting alone.
• Scoring is a direct comparison, so this case needs no rubric and no judge model.

Try it out

Before collecting more data, try the schema on one case. You should be able to produce the judgement with the information the case contains.