Execute: Run the System, then the Judge

Recap: what you bring into this chapter

Execution means running a design you have already settled. Before you start, you should have:

• Definition of "good": A concise, stakeholder-agreed list of the specifications your product must show, from Define: What "Good" Looks Like
• Judgement design and metrics: A judgement design for each specification, summarised into a basket of metrics with acceptance criteria, from Judgement Design
• Evaluation set: A trustworthy, representative evaluation set to run the judgement design over, from Evaluation Set Design

If any of these is missing, settle it in that chapter first, then come back here to run the evaluation.

With the design settled and the resources in place, you are ready to run the evaluation. On paper, execution sounds mechanical: run every test case through the system, put each artefact in front of the judge, and collect the judgements.

In practice, however, two problems might arise that can undermine the trustworthiness of the results:

1. Reliability: Randomness might make the same input produce different outputs, and consequently different judgements each run. If one run says pass and the next says fail, which can be trusted to be the truth?
2. Observability: Your system and judges might behave like black boxes; if a surprising result appears, are you able to tell what caused it? A surprising result is only useful when you can trace it back to what caused it.

We will explore each challenge in turn, and the respective dials to turn and manage them.

---

Reliability: the consistency dial

Execution is only reliable when the same input consistently produces the outputs and judgements across repeated runs. Variation can arise from either side of the pipeline; check below to decide whether it is a concern for you.

Quick check

Do you need to actively manage reliability?

Input

Systemblack box

Is your system stochastic?Applies if any output, be it intermediate or final, can change between runs.

Artefactoutput + context

Evaluation designblack box

Is your evaluation design stochastic?Applies if the same artefact can receive different judgements across runs. This typically applies for LLM-judged evaluation designs.

Judgement

A yes to either question means you might struggle to trust a single run; take note of best practices for managing reliability in the pages that follow.

Wherever it comes from, variation takes one of two forms on both sides of the pipeline:

	Avoidable variation	Inherent randomness
What it is	Noise that creeps in from inconsistent execution	Randomness built into how a component works
What to do	Minimise it	Manage its degree
Examples	Output format drift despite fixed instructions (system) One rubric, applied unevenly (evaluation)	LLM text generation (system) LLM-as-a-judge scoring (evaluation)

Watch for avoidable variation

The two forms also differ in how they show up. Inherent randomness is usually known upfront, because it arrives with the technology you chose, such as an LLM system or an LLM judge. Avoidable variation tends to creep in unnoticed, so it pays to actively look for it.

Observability: the depth dial

An execution is sufficiently observable when you can explain judgements from the artefacts that produced them, and as an extension, explain the artefacts from the relevant inputs. Check below to decide whether this is a concern for you.

Quick check

Are your runs sufficiently observable?

Input

Systemblack box

Artefactoutput + context

Do you understand how artefacts are produced?This only applies when you are unable to trace surprising artefacts back to how the inputs produced them.

Evaluation designblack box

Judgement

Do you understand how judgements are produced?This only applies when you are unable to trace surprising judgement scores back to the artefacts that were judged.

A yes to either question implies you are not capturing sufficient traces; read on for best practices to improve observability within your runs.

Always capture a baseline

You cannot evaluate what you cannot observe: if nothing records what the system did, a surprising score has nowhere to lead and a failure cannot be diagnosed. Capturing a baseline level of observability on every run is strongly recommended, enough to explain any result, even when resources are tight.

Beyond that baseline, more depth gives 4. Improve more to work with but costs more (time, compute, and human effort), so spend the rest as the dimension × resource trade-off allows.

The dimension × resource trade-off

Turning either dial up costs resources: more repeats mean more compute and spend, and deeper traces mean more capture, storage, and review effort.

Because both dials draw on the same finite budget, the more rigour you buy the more you spend. How far up that line you sit depends on the system, the evaluation objective, and the stakes:

• Fast iteration loop (early development): stay cheap and re-run with more rigour once something looks promising.
• High-stakes evaluation (a launch gate or safety check): spend beyond the everyday budget, because a wrong conclusion dwarfs the compute.

Reliability and observability draw on the same budget, and buying more of either costs more; set how far up the line you sit by the stakes.

---

How this chapter is organised

Execution splits into two phases, in the order you work through them. First you run the system, tuning reliability and observability on the system side and launching the full run. Then you run the judge over the behaviour it captured, tuning the same two dials on the evaluation side.

1. 13a. Run the systemHow do we run the system and capture evidence we can trust, then launch the full run?
2. 23b. Run the judgeHow do we judge the captured behaviour consistently and explainably?

What you will have by the end

• Results you can trustReliability and observability tuned to match the stakes, with the run itself kept within budget