Offline Evaluation
Previously, online evaluation helped surface failure modes in production that may warrant further attention.
This is where offline evaluation comes in. It provides a controlled setting to investigate these failure modes further, and to design and validate improvements, using a selected evaluation set through the functions below.
- Component testingIs this part doing its job, and does the end-to-end result still hold?
- BenchmarkingWhich model, prompt or configuration performs best on the same cases?
- Regression testingDid this change break behaviour that previously worked?
Every offline evaluation rests on three components you built in earlier chapters for a single pass of the evaluation loop before deployment.
| Component | What it is |
|---|---|
| Execution setup | The versions and settings each run uses, with its results and traces |
| Evaluation set | The cases you test against |
| Evaluation design | The judges, metrics and acceptance criteria you apply |
The existing product requirements that ground online evaluation also ground these offline evaluation components. Production findings tell you where to focus; what you test and how you read the results stay anchored to those shared requirements.
Making offline evaluation continuous
Evidence that one system version meets baseline requirements does not guarantee that subsequent versions will, so offline evaluation must continue as a routine part of assessing proposed changes before they reach production.
We identify two principles that help keep this repeatable, relevant and sustainable.
Principle 1: Build a reusable workflow and automate recurring runs
Maintain a reproducible evaluation workflow
In the earlier chapters, we discussed building the underlying components of offline evaluation. Now, keep the evaluation basis consistent so that result differences can be attributed to the candidate system, rather than changes in evaluation design or execution.
Run in an environment separate from production
Run offline evaluations in a staging environment set up with the candidate version, not on the live system. This keeps test traffic out of real logs, analytics and user records. It also lets you stop, reset and rerun a run without affecting users.
Record each run and its evidence
Preserve the system version, evaluation-set and judge versions, execution settings, case-level results and supporting traces. This makes it possible to compare a candidate with an approved baseline and investigate cases that improve or regress. Missing or failed cases should remain visible rather than being treated as successful results.
Automate recurring work where possible
Connect the workflow to relevant development events, such as commits, pull requests or releases. Automate offline evaluation runs on the relevant levels, and record the results in a way that streamlines diagnosis.
Choose what to run and how often according to the change, risk, cost and decision involved. For example, small, focused suites can be automated to run on every pull request, but larger changes warrant a more thorough end-to-end offline evaluation before release.
Principle 2: Keep the evaluation design & set current and relevant
Preserve important regression coverage
Retain selected, representative cases covering core workflows, previously fixed failures, known edge cases and behaviour you expect future versions to preserve.
Incorporate new findings into the regression suite
Online evaluation may surface failure modes or scenarios that your existing offline suite does not cover. Add representative cases for these new findings so future system changes are checked against them as well.
Worked example: A change to the appeals assistant
A change to the summarisation instructions aims to preserve income figures more accurately.
- Reusable workflow, run automatically. Opening the pull request runs the focused suite on the usual basis (the cases where income figures were misread, plus previously passing summaries) and records the results against the approved baseline, so the change can be compared case by case before release.
- A suite kept current. After release, online evaluation surfaces a payslip format the summary misreads; that case is added to the versioned suite, so every later version is checked against it.
This chapter's takeaway is more about understanding concepts than creating a new item. Looking back at What you will have by the end, you should now be able to explain:
- Why continuous evaluation is important for production systems
- How the two modes, online and offline evaluation, complement each other
- How to keep the evaluation loop repeatable, relevant and sustainable
The loop does not end here: live production data keeps surfacing new findings, offline evaluation keeps validating changes, and each pass sharpens both the system and the evaluation design.
To understand implementation details further, the MLOps Playbook picks up where this chapter leaves off.
Was this page helpful?