Online Evaluation

---

How online evaluation differs

As covered in 1. Define, every evaluation is grounded in a purpose and in the product requirements that define it. Online evaluation is one particular case of this.

Once a product is deployed, its requirements are largely settled. Online evaluation therefore places greater emphasis on whether the system delivers its intended outcomes for users in practice, not only whether it meets its acceptance criteria in a controlled setting.

What matters in production

In production, it often helps to focus on two areas:

Core technical requirementsThe critical requirements the product must keep meeting in use.Core technical metricsBusiness outcomesThe benefits the product is intended to deliver for its users.Outcome metrics

Core technical metrics

For those who did not read the previous chapters

The core technical metrics referenced here can be derived from the evaluation design you built before deployment.

These metrics stay graded in production: you track the level and trend of each one on live traffic, such as an accuracy rate or processing time, rather than only watching for outright failures.

Not every metric should run on live traffic, though. Each one adds cost at production scale, and too many alerts bury the ones that matter, so carry over only a focused subset to monitor continuously. Metrics you leave offline are not discarded: they stay in your evaluation design and are still measured in offline evaluation, for example in regression runs before each release.

These guidelines can help you decide what to carry over, and what to leave offline:

Monitor known requirements and failure modes

Carry overMetrics guarding mandatory requirements, and checks behind frequent or high-impact failures

Leave offlineMetrics and checks tied to low-priority requirements or minor failures

Keep judges that rely on production evidence

Carry overBehaviour that can be judged from production traces

Leave offlineBehaviour that can only be judged with reference context

Prefer judges that are consistent & inexpensive to run at scale

Carry overJudges like rule-based, or some model-based, that incur low cost and are easily validated

Leave offlineJudges that are expensive to run, or require human validation

Carry over only what is core

Not every metric is needed in production. Select only the core requirements of the live product that are worth observing on real traffic.

For example, the appeals assistant should keep tracking requirements such as recommendation correctness and processing time, but leave detailed component checks to offline evaluation.

Outcome metrics

While technical metrics tell you how the system behaves, outcome metrics tell you what it achieves for users. In 1. Define, we examined "good" from different perspectives.

Outcome metrics speak mainly to the business perspective, which you can only observe once real users are involved.

For example:

• Time or effort saved: Does the system reduce the time or work users spend, such as staff-hours per case?
• Adoption and continued use: Where relevant to the product objective, do users keep choosing to use the feature?
• User satisfaction and trust: Do users find the system useful, and rely on it appropriately?
• Value relative to cost: Does the benefit justify the cost of running the system?

When to add outcome metrics

Add outcome metrics when your question goes beyond whether the system produced a good response. They are especially useful when:

• A product requirement concerns real-world impact, such as saving time, resolving cases or improving productivity.
• A release or product decision depends on user value, not just output quality.
• You need to understand a trade-off, such as whether higher cost or latency is justified by better user outcomes.

Interpret outcome signals with care

Treat outcome metrics like any other requirement in your product specification, but bear in mind that some signals can be biased or unrepresentative, for example when only some users leave feedback.

Read them alongside the core technical metrics above. A faster completion time is not an improvement if accuracy or safety falls below the acceptance criteria.

Methodologies for online evaluation

Each of these approaches uses live data to answer a different question. They complement each other and can be used together.

Evaluate production interactions

Score or review sampled live traces.

Use when...You need to know whether the deployed system is meeting its requirements.

Tells you...Whether defined acceptance criteria are met, and any new or existing failure modes appear.

Collect user feedback and behavioural signals

Ask users to rate/comment on outputs, and track any interactions they have with the system.

Use when...The priorities are mainly on user or product value.

Tells you...Inform outcome metrics that might be part of the product requirements.

Run online experiments

Use statistical methods like A/B testing to evaluate product performance on live traffic.

Use when...You need to test changes on a live environment.

Tells you...Estimates effect sizes for outcome metrics due to system changes.

Monitoring online evaluation over time

A deployed system rarely stays as it was at launch. As more and more varied users rely on it,

• Inputs can drift. Different user groups ask new kinds of questions, or use the system in ways you did not anticipate.
• The range of outputs grows. A wider spread of inputs produces a wider spread of responses, including rare cases your evaluation set never covered.
• Concepts may drift. The relationship between inputs and correct outputs can change over time, leading to a decline in system performance.

These effects compound at scale and can quietly erode what matters in production long before anyone reports a problem. This is why monitoring and alerting are crucial to performing online evaluation well.

Systems that learn from feedback can drift too

If your system uses production feedback to update itself, for example through reinforcement learning from human feedback (RLHF), the model can change between releases, causing model drift, where model behaviour changes.

This can be a source of unexpected regressions and new failure modes, so online evaluation is especially important for these systems.

What to monitor

We recommend monitoring the following two areas:

• Your metrics: Core technical metrics and outcome metrics we have previously discussed.
• Input drift: Simple indicators such as request types, input lengths or document formats, to tell whether a change comes from the system or from its expected users.

Alert only on what needs action

Set alerts on changes that cause mandatory acceptance criteria to fail, so that the right person can review the traces behind them. Findings that warrant a closer look then move into offline evaluation.

Further details on implementation

For implementation details on online evaluation, refer to the MLOps Playbook.

How online evaluation informs offline evaluation

Through monitoring and alerting, online evaluation can surface unexpected failure modes, and these findings can be taken into offline evaluation for a more controlled investigation.

When you take a finding into offline evaluation, we recommend carrying over:

Carry over	What it includes	Why
The traces	Inputs, outputs and intermediate steps, such as retrieved context and tool calls	To reproduce the failure mode in a controlled setting
The versions and settings in use	The system version and configuration at the time	So you investigate what actually ran
What went wrong	The requirement or failure mode the finding relates to, and the evidence behind it, such as a judge verdict, a user correction or feedback	To know what the investigation is testing
The kind of case	The input type, workflow or user group the case belongs to, and how often similar cases occur	To choose representative test cases and prioritise
Your existing objectives and requirements	The product requirements that ground online evaluation	So the offline investigation is held to the same standard

Worked example: Carrying a finding from the appeals assistant into offline evaluation

The appeals assistant reads supporting documents, summarises case information and supports an officer's recommendation. Monitoring flags a sustained drop in the accuracy of income figures in summaries, and officers are correcting more summaries than usual.

Before investigating offline, the team gathers:

1. The traces: The flagged cases, with the uploaded documents, the extracted text, the summary and the officer's correction.
2. The versions and settings in use: The system and prompt versions running when the drop began.
3. What went wrong: Income figures misread in summaries, which breaches a mandatory requirement, evidenced by the judge's verdicts and the officers' corrections.
4. The kind of case: Most failures involve a payslip format the evaluation set did not cover, which tells the team which cases to sample for offline testing.
5. Existing objectives and requirements: Recommendation correctness still governs the investigation; the goal is not just to make these payslips pass.

A misread payslip is a reason to investigate offline; it does not by itself establish the cause.