Skip to main content

Monitor: Productionalising and Scaling the Evaluation

Recap: what you bring into this chapter

By now, you should have finished a single pass of the evaluation loop. You should have:

  • Definition of "good": A concise, stakeholder-agreed list of the specifications your product must show, from Define: What "Good" Looks Like
  • Judgement design and metrics: A judgement design for each specification, summarised into a basket of metrics with acceptance criteria, from Judgement Design
  • Evaluation set: A trustworthy, representative evaluation set to run the judgement design over, from Evaluation Set Design
  • Completed results: A score per case on each metric, with the traces behind them, from Execute: Run the System, then the Judge
  • Diagnosis & improvements: A structured way to diagnose failures and validate improvements from the results, from Improve: Diagnose and Refine

From one pass to continuous evaluation​

Now that you have completed one pass of the core evaluation loop, the loop does not end there. Each round of 4. Improve leads to new changes in either the system or evaluation design, or both, prompting yet another round of evaluation.

Beyond the core evaluation loop, once you finish building and deploy your product to a live environment, what happens?

We refer back to the evaluation approach we established at the start:

CORE EVALUATION LOOP · REPEATS AS YOU BUILDOUTER LOOP · AFTER LAUNCHafter launchfeeds back5MonitorAdapting evaluation design to be continuousin production environments.1DefineDefine what “good” looks like: intended tasks,expected outcomes and application requirements.2DesignDesign how to measure “good” and buildrealistic, trustworthy evaluation sets.a.Judgement DesignJudgement methods,required evidence,acceptance criteria.b.Evaluation SetDesignBuild realistic andtrustworthy evaluationsets.3ExecuteRun the test suite against the evaluation designconsistently and capture execution traces.a.Run the systemExecute the system andcollect artefacts andrelevant traces.b.Run the judgeRunning the evaluationdesign to assess capturedbehaviour and outcomes.4ImproveDiagnose test results and extract insights to guideapplication improvements and future evaluations.
Recap: the evaluation loop that grounds our approach.

Once your product goes live, several considerations grow increasingly important:

  • Real users depend on it. Failures and regressions now have real-world consequences.
  • The system continues to evolve. Issues are fixed, features are added, and new versions are released.
  • The evaluation evolves alongside it. Criteria are refined, judges are adjusted, and new test cases are drawn from real-world usage.

The challenge is no longer just running the evaluation, but keeping the evaluation working for an evolving product in deployment.

We can frame this challenge as:

How can we turn our continuous evaluation loop into something repeatable, relevant and sustainable?

To answer this, we can ground the concepts of continuous evaluation in these three themes:

Repeatable

Same evaluation loop repeated with compatible versions of each part to produce labelled results.

Systemv2v1v3
Evaluation designv1v2
Evaluation setv3v2v4
Specified configurationsDiagnosed experiment results
Relevant

Evaluations stay core to the product's development and usage.

  • Updates as product requirements change.
  • Self-improving; becomes more targeted to the product over time.
  • Keeps diagnoses and improvements actionable.
Sustainable

Evaluation loop stays manageable alongside product growth.

Workload grows
  • Evaluation set
  • Product requirements
  • Evaluation frequency
Overheads grow more slowly
  • Cost
  • Turnaround time
  • Human effort

Continuous evaluation at scale​

Continuous evaluation operates in two complementary settings: online evaluation and offline evaluation.

The distinction is not about choosing one over the other, but about using both together to answer questions central to an already deployed system.

Online evaluation

Assesses how the deployed system performs in actual use.

How does it behave in use?

Offline evaluation

Assesses system versions and proposed changes under controlled conditions.

Is this version ready to release?

Part of a wider MLOps practice

This chapter explores these concepts briefly to provide a foundation for understanding why continuous evaluation is important in a production environment. For specific implementation details and resources, refer to our MLOps Playbook.

How this chapter is organised​

The chapter covers the two settings of continuous evaluation in turn: online evaluation on the live system, then offline evaluation in a controlled environment before each release.

  1. Online evaluationHow do you evaluate the system under real-world usage?
  2. Offline evaluationHow do you run evaluations repeatably, in a controlled environment, before future releases?

What you will have by the end​

A better understanding of continuous evaluation, covering:

  • Why it mattersWhy continuous evaluation is important for production systems
  • Offline and online modesThe different modes of continuous evaluation and how they complement each other
  • Repeatable, relevant, sustainableHow to make continuous evaluation repeatable, relevant and sustainable
Last updated on

Was this page helpful?