Run the System

In Judgement Design, you decided which behaviours or outputs of your system are the subjects of judgement. In Evaluation Set Design, you decided which cases and conditions test them. Running the system puts those cases through the relevant parts of the system and captures the artefacts the judge will need.

Depending on the subject of judgement, the part you run may be a single component, a subsystem or the whole application. For simplicity, this page calls whichever part you run the system.

Two dials, both introduced earlier, decide how far you can trust what comes out:

• Reliability: The system behaves consistently, or varies only in controlled ways.
• Observability: You can see how each output was produced.

This page covers both from the system side, then what to run and how to run your full evaluation set.

---

Managing system reliability

We need to ensure that the system behaves consistently, or varies in a controlled manner. This is crucial for ensuring our evaluation results are sufficiently meaningful for diagnosis and coming up with improvements.

Deterministic or stochastic?

Some systems are deterministic: given the same input under the same conditions, they should give the same output every time. Others are stochastic by design, such as an LLM that samples its responses, and can legitimately give different outputs for the same input. In both cases, you want differences in output to come from the system's intended behaviour, not from uncontrolled changes in how it was run.

The principles of minimising avoidable variation applies to every system, and managing inherent randomness applies to stochastic ones.

Minimising avoidable variation

1. Configure the system for what you are testing. Depending on whether you are evaluating a component, subsystem or the whole system, decide which dependencies, data sources and settings should be included, fixed or simulated. Ensure the setup reflects the behaviour you intend to evaluate.
2. Keep configurations consistent across comparable runs. Fix the relevant component versions, settings and dependencies, changing only what you deliberately intend to test. Otherwise, differences in results may come from changes in the setup rather than the system behaviour you are evaluating.
3. Compare across consistent evaluation sets. Hold the evaluation set, its ordering, and any splits fixed between runs. A result is only comparable to another when measured on the same inputs.

Notes: Complex systems

As systems grow in complexity, more components and dependencies introduce additional sources of variation, making it harder to identify what caused differences between runs.

• Fix tool and dependency versions. Pin the versions and timeouts of the external tools, libraries, and APIs the system calls. A tool that behaves differently between runs is a hidden source of drift.
• Constrain the output format. Where the task allows, require a fixed schema so that any variation shows up in substance, not format, and parsing stays deterministic.

Managing inherent randomness

1. Set randomness to match the system's purpose. Keep it low where the system should give the same answer every time, and allow more where variety is the point. Evaluate with the settings the system uses in real use, or you measure behaviour your users never see.
2. Use seeds to reproduce, not to measure. A seed is a setting that makes a random run repeatable, which helps when you need to rerun or debug a specific output. A fixed seed also hides variation, so leave it free when the range of outputs is what you want to see.
3. Repeat runs when variability matters. One run of a stochastic system shows one possible output. Running each case several times, each sampled independently, shows the range of outputs and gives a more precise estimate of how the system performs. Repeats do not make the system itself more reliable, and a deterministic system needs only one run per case.

Notes: LLM-based systems

An LLM-based system is usually stochastic; the same input can produce different outputs. However, you can still manage the degree of randomness through the model's sampling parameters. The most common ones:

• temperature: How much randomness the model uses when picking the next word.
  • Scale: commonly 0–1 or 0–2, depending on the provider; lower favours the most likely words, higher allows less likely ones.
• top_p: Limits choices to a changing shortlist of likely options.
  • Scale: 0–1; lower leaves fewer words to choose from, higher leaves more.
• top_k: Limits choices to a fixed number (k) of the most likely options.
  • Scale: whole number from 1 onwards; top_k of 1 always picks the single most likely word.

Not every model or provider exposes all of these settings, or a seed. Check what yours supports.

Low temperature is not a guarantee

A temperature of 0 makes outputs far more consistent, but it does not guarantee identical outputs. Small differences in how the provider runs the model can still change them. Check consistency on a sample of repeated runs rather than assuming it.

One at a time

Change one sampling parameter at a time. Move several at once and you cannot tell which one caused the shift in output.

Match settings to the system's purpose. The examples below use a generic client, as parameter names and ranges vary by provider.

An HR chatbot (a support chatbot answering questions from a specified knowledge base) should aim to give the same answer every time, so pin sampling low:

# Consistency-oriented: as consistent an answer as possible
response = llm.generate(
    model="model-name-2026-01-15",  # pinned version, not a floating alias
    temperature=0,                  # favour the most likely words
    messages=messages,
)

One run per case is often enough here, with a few repeats on a sample of cases to confirm that outputs are stable.

A simulation system (generating varied user personas, exploring different dialogues an agent might face) needs that variety, so keep sampling loose and run several independent attempts:

# Stochastic: variety is the point
responses = [
    llm.generate(
        model="model-name-2026-01-15",  # still pinned
        temperature=1,                  # sample freely for diverse outputs
        messages=messages,              # no fixed seed, so each attempt can differ
    )
    for _ in range(5)                   # 5 independent attempts
]

Here, the range of outputs is what you evaluate, so capture each attempt separately.

How many repeats?

Repeats matter most when the system is stochastic and its variation matters to your evaluation.

Repetition is a near-linear cost multiplier: ten repeats is roughly ten times the compute. There is no fixed number: more repeats give a more precise picture of how the system behaves, but in most setups the count is capped by the resources you can spare. This follows the dimension × resource trade-off.

Scale according to the stakes and available resources.

Managing system observability

Which signals are worth capturing depends on your system and the evaluation objectives you defined earlier.

Captured signals serve two purposes. Some form part of the artefact that the judge assesses, such as a tool call that is itself the subject of judgement. Others help you explain a surprising judgement later and find where a failure started.

The following are useful defaults across most systems:

Signal	What it helps reveal	Capture when...
Intermediate, unseen workings	How the system arrived at its output	Improving complex systems where you need enough detail to locate where failures originate
Errors, retries, and timeouts	Failures that occurred during execution but may not surface in the final output	Improving a system with external tools, services, or operations which can fail or recover silently
Latency and resource usage	Whether slow or costly execution coincides with poor outcomes	Comparing products on efficiency, or supporting decisions where operational cost and feasibility matter

In practice, these signals are captured as a trace for each execution, usually viewed in a tracing tool. When a case runs more than once, give each attempt its own trace, labelled with the case and attempt, so every output can be traced back to the run that produced it.

Worked example: A captured trace

A captured trace for one attempt at a GrantsAssist case might look like this:

{
  "trace_id": "tr-0423-1",
  "case_id": "grant-0423",
  "attempt": 1,
  "status": "completed",
  "input": "Am I eligible for the Startup SG Founder grant?",
  "output": "You are likely eligible: as a first-time founder who is a Singapore citizen, you meet the core criteria...",
  "system": {
    "version": "2.3.0",
    "model": "model-name-2026-01-15",
    "temperature": 0.3
  },
  "tool_calls": [
    {
      "tool": "eligibility_search",
      "args": { "scheme": "Startup SG Founder" },
      "status": "ok",
      "latency_ms": 410,
      "response": {
        "citizenship": "Singapore citizen or PR",
        "first_time_founder_required": true
      }
    }
  ],
  "retries": 1,
  "errors": [],
  "latency_ms": 7320
}

Each field is a signal from the table above: case_id and attempt tie the trace to one execution, tool_calls shows how the answer was reached, retries and errors surface failures that the final output would hide, and latency_ms flags slow cases.

Notes: LLM-based and agentic systems

LLM and agentic systems are often opaque by nature. Most of the signals below record what the system actually did, such as which tool it called and what came back. Model-provided reasoning is different: it is text the model generates about its own process, so treat it as supporting evidence, not a complete record of how the model reached its output.

Signal	What it helps reveal	Capture when...
Tool calls and responses	Whether the system chose the right tool and used it correctly	System calls external tools or services to produce its output
Routing/orchestration decisions	Which model, sub-agent, or workflow path the system chose	Improving a system that dynamically chooses between different LLMs for different tasks
Model-provided reasoning	The reasoning text or summary the model outputs, where the model or provider makes it available	System relies on extended reasoning, and you need extra context to diagnose decisions
Memory operations	What information was stored, retrieved, updated, or carried forward across turns	Persistent system that executes over multiple interactions
Agent state	What goals, task status, or working information the agent held at a given point	System maintains state that influences subsequent decisions
Execution trajectory	How actions, observations, and state changes unfolded over the full run	Evaluating longer agentic workflows where the sequence of decisions matters

For an agentic run, this extends to the trajectory: each step's action and observation, plus any reasoning the model provides.

[
  {
    "step": 1,
    "reasoning": "I need the applicant's citizenship before I can check eligibility.",
    "action": { "tool": "profile_lookup", "args": { "field": "citizenship" } },
    "observation": "permanent_resident"
  },
  {
    "step": 2,
    "reasoning": "PRs are eligible for this scheme; confirm the funding cap.",
    "action": {
      "tool": "eligibility_search",
      "args": { "scheme": "Startup SG Founder" }
    },
    "observation": { "pr_eligible": true, "funding_cap": "S$50,000" }
  }
]

Don't capture everything

You will not need every signal for every evaluation. Capture only what is necessary as it incurs additional resources, see dimension × resource trade-off.

Running the full evaluation set against the system

After understanding how to ensure reliable runs and how much to observe, you are ready to run your full evaluation set.

The goal of this stage is a complete set of captured artefacts: for every attempt at every case, the output, its trace and whether it completed.

Decide which version to run on

Before running the evaluation, decide which version of each relevant system component to use. This should align with what your evaluation set is designed to test.

Include the system layers that shape the behaviour being judged, unless deliberately testing a component in isolation.

• ContextProfiles, history, permissions and live data.Set the user profile, history and permissions for each test case to match a real user, and record them in the case.
• Logic & integrationsRouting, retrieval, tools, data sources and business rules.Connect each test case to the same tools and data sources as the version under test. Fix a step's output in the case only when you judge the step after it on its own.
• SafeguardsInput and output checks, filters and fallbacks.Leave them on for every test case, as real users meet them, unless your evaluation design scopes them out.
• ConfigurationPrompts, thresholds, parameters and feature flags.Run every test case on the exact configuration under test, and keep it fixed for the whole run.
• ModelThe LLM, classifier or rules engine at the core.Pin the model version and settings, and run every test case on them.

Which version to run follows your evaluation objectives.

If it is...

• Improving the system,Your candidate change, beside a fixed baseline version to compare it against.
• Comparing products,Each candidate, such as different models, vendors or configurations, under identical settings.
• Supporting a decision,The exact release candidate, with its shipping configuration, in a staging environment that mirrors production.

Best practices on optimising runs

A few practices keep the run from stalling, wasting resources or hiding failures.

Before the run

• Dry-run 10–20 cases. Read the outputs and traces in full to catch parsing errors, missing fields and truncated answers early.
• Estimate cost and time. System calls are cases × repeats. If you budget for the judge at the same time, add its calls separately, as they often differ in cost and speed. Over budget? Cut repeats or trace depth, following the dimension × resource trade-off.

Worked example: Estimating a run

GrantsAssist is evaluated on an evaluation set of 1,000 test cases, with 5 repeats each. The team also budgets for an LLM judge that assesses each captured output once. The dry run averaged $0.03 and 8 seconds per system call, and $0.01 and 3 seconds per judge call:

	Calculation	Estimate
System calls	1,000 cases × 5 repeats	5,000
Judge calls	5,000 outputs × 1 judgement	5,000
Cost	(5,000 × $0.03) + (5,000 × $0.01)	~$200
Time, one call at a time	(5,000 × 8 s) + (5,000 × 3 s)	~15 hours
Time, 10 calls in parallel	15 hours ÷ 10	~1.5 hours

If $200 is over budget, dropping to 3 repeats cuts both system and judge calls to 3,000 each, bringing the total to ~$120. Whether that is acceptable depends on the stakes.

During the run

• Automate whenever possible. Script the loop so each case runs and is captured without manual steps. Hand-running cases does not scale and introduces inconsistency.
  • Prefer an API if the system can expose one.
  • Store artefacts apart from judgements if you judge as you go, so you can rerun the judge alone.

No API?

Drive the UI with a browser automation library such as Playwright, and keep the run resumable. Do take note that this is more brittle than an API, and UI changes can break the script, so proceed with caution.

• Retry harness failures, record system failures.
  • Harness failures, such as rate limits from sending cases too fast, say nothing about the system: retry them.
  • System failures, such as errors, timeouts or failed tool calls, may be genuine: record them as that attempt's outcome. If the system retries on its own in real use, let it do so here too.
• Checkpoint as you go. Write each result as it completes, so a crashed run resumes instead of restarting.
• Set stop rules upfront. For example, halt if harness errors exceed 5%, system timeouts exceed 10%, or spend passes the budget. A spike usually points to a setup problem worth fixing before you continue.

Consider a sandbox for risky actions

If the system takes real actions, such as sending messages, writing records or calling external services, or if you are red-teaming with harmful inputs, run it in a sandbox. A sandbox is an isolated copy of the system with mock tools and synthetic data. It lets you test these cases without real side effects. For how to generate red-team inputs, see Adversarial and red-team prompt generation in the Responsible AI Playbook.

Here is an example of running the full evaluation set through GrantsAssist, showing how the above practices come together.

Worked example: Running the full evaluation set

GrantsAssist (a grants-eligibility assistant that calls a retrieval tool) is evaluated as a whole, so each case's input is a user's question. It runs with temperature=0.3 in real use, so the same question can get different answers. To see how much they vary, the team runs each of the 1,000 cases 5 times with no fixed seed, giving 5,000 independent attempts.

The run script. Each attempt is one scripted call, stored as soon as it completes:

import grantsassist  # the whole assistant, behind an API

def run_attempt(case, attempt):
    result = with_retry(          # retries harness errors only, such as rate limits
        grantsassist.ask,
        case["input"],
        version="2.3.0",          # pinned system version
        temperature=0.3,          # as in real use, with no fixed seed
        capture_trace=True,       # tool calls, errors, latency
    )
    return {
        "case_id": case["id"],
        "attempt": attempt,
        "status": result.status,  # "completed", "timeout" or "error", kept as is
        "output": result.output,
        "trace_id": result.trace_id,
    }

for case in eval_set:
    for attempt in range(1, 6):            # 5 independent attempts per case
        store(run_attempt(case, attempt))  # checkpoint: write as each completes
        if timeout_rate() > 0.10:          # stop rule
            raise StopRun("timeouts exceeded 10%")

What happened

• Dry run. Of 15 cases, two outputs failed to parse because the script read the wrong response field, and one trace was missing its tool-call log. Both were harness problems, fixed before the full run without changing the system.
• Estimate. About $150 for 5,000 system calls, plus about $50 for judging, came in within budget, so repeats stayed at 5.
• During the run. The script hit the provider's rate limit twice; those calls were retried and not counted against the system. Separately, 31 attempts (under 1%) timed out inside the retrieval tool. These were recorded as system timeouts, not retried.

Captured artefacts. Each attempt is stored as its own record, linked to its trace. Three of the five attempts for one case:

[
  {
    "case_id": "grant-0423",
    "attempt": 1,
    "status": "completed",
    "output": "You are likely eligible: as a first-time founder who is a Singapore citizen, you meet the core criteria...",
    "trace_id": "tr-0423-1"
  },
  {
    "case_id": "grant-0423",
    "attempt": 2,
    "status": "completed",
    "output": "You may be eligible, depending on whether this is your first company. Is it?",
    "trace_id": "tr-0423-2"
  },
  {
    "case_id": "grant-0423",
    "attempt": 4,
    "status": "timeout",
    "output": null,
    "error": "eligibility_search timed out after 10 s",
    "trace_id": "tr-0423-4"
  }
]

Result: 5,000 records, each with its status, output and trace, ready for the judge. Attempts at the same case do not always agree, as attempts 1 and 2 show; that variation is evidence in its own right. Failed attempts stay in the set with their reason, rather than being dropped or retried away.