Skip to main content

Improve: Diagnose and Refine

Recap: what you bring into this chapter

Diagnosis works only if the earlier stages are in place. You should already have:

  • Definition of "good": A concise, stakeholder-agreed list of the specifications your product must show, from Define: What "Good" Looks Like
  • Judgement design and metrics: A judgement design for each specification, summarised into a basket of metrics with acceptance criteria, from Judgement Design
  • Evaluation set: A trustworthy, representative evaluation set to run the judgement design over, from Evaluation Set Design
  • Completed results: A score per case on each metric, with the traces behind them, from Execute: Run the System, then the Judge

If any of these is missing, settle it in that chapter first and come back after.

So far, your results only tell you what happened. We now need to:

  • Diagnose: Why did the system perform this way, and what is going wrong behind each gap?
  • Improve: What change(s) can we make to address that, and do they actually help?

Failures are rarely one-offs; most are instances of a failure mode: a distinct, recurring way the system goes wrong, such as misreading income figures from scanned documents.

Not just your system

Both questions can apply to the evaluation design as well, not only the product under test. A surprising result sometimes comes from a flawed evaluation rather than a flawed system.

An iterative process​

Every improvement is a change, and a change can only be trusted once it has been re-evaluated.

That is what makes this the most iterative stage of the process: each change can send you back to any stage in the loop.

CORE EVALUATION LOOP · REPEATS AS YOU BUILDOUTER LOOP · AFTER LAUNCHafter launchfeeds back5MonitorAdapting evaluation design to be continuousin production environments.1DefineDefine what “good” looks like: intended tasks,expected outcomes and application requirements.2DesignDesign how to measure “good” and buildrealistic, trustworthy evaluation sets.a.Judgement DesignJudgement methods,required evidence,acceptance criteria.b.Evaluation SetDesignBuild realistic andtrustworthy evaluationsets.3ExecuteRun the test suite against the evaluation designconsistently and capture execution traces.a.Run the systemExecute the system andcollect artefacts andrelevant traces.b.Run the judgeRunning the evaluationdesign to assess capturedbehaviour and outcomes.4ImproveDiagnose test results and extract insights to guideapplication improvements and future evaluations.
The dashed lines from Improve lead back to every earlier stage: a change here can send you to any of them.

This chapter guides you through a recommended structure for a single pass of diagnosis and improvement, which you can adapt and reuse on every pass after it.

No one-size-fits-all approach

The structure here is a recommended starting point. Adapt it to your system, your evaluation design, and the resources you have.

How this chapter is organised​

The chapter walks through one pass of the loop, in order: assess performance against your acceptance criteria, analyse the failures behind the gaps into failure modes, then design and validate the improvements that follow.

  1. Assessing performanceWhich acceptance criteria did the results meet, and where are the gaps?
  2. Analysing failuresWhat is going wrong behind the gaps, and why?
  3. Designing improvementsWhat should you change, and how do you confirm it helped?

What you will have by the end​

  • A failure taxonomyA lean, comprehensive catalogue of your system's failure modes, each tied to a cause
  • Validated changesA set of changes validated to address those failure modes
Last updated on

Was this page helpful?