Designing & Validating Improvements

From analysing failures, you have a lean, comprehensive failure mode taxonomy.

A taxonomy is only useful once each failure mode has a change that addresses it. This page covers that in two steps: designing candidate improvements, then validating that they work.

---

Designing improvements

Diagnosis identifies failure modes, but not every failure mode has a clear fix. Some fixes are obvious, some take more thought, and some failure modes may have no fix at all.

The goal of this stage is to propose a list of candidate changes that could plausibly address every failure mode, given the available resources and constraints.

There is no single correct way to design improvements, but the structure below can guide the process:

1Propose candidate changes

Plausible changeWhat to change, and why the evidence suggests it addresses the failure mode.

Expected outcomeThe observable improvement if it works, tied to the affected behaviour and product requirements.

Expected effortWhat it takes to build and validate, including any ongoing costs.

↓

2Consolidate

Merge duplicate proposalsCombine proposals that describe the same change, keeping the link to every failure mode each one addresses.

↓

3Prioritise & select

A manageable set of changesChosen by expected outcome and effort, within your resources, with the rest deferred or flagged for further investigation.

Worked example: Designing improvements for the appeals assistant

Start from the taxonomy produced in analysing failures: a misread numeric figure (summary, 48%), the wrong rules system called (tool call, 27%), and a document missed entirely (summary, 15%).

Propose a candidate for each, defined by the plausible change, the expected outcome and the expected effort:

Failure mode	Plausible change	Expected outcome	Expected effort
Misread figure (48%)	Instruct the summariser to copy income figures verbatim, with no rounding	Fewer rounding discrepancies, and fewer wrong recommendations near the eligibility threshold	Low: a prompt change and a re-run
Misread figure (48%)	Add a structured extraction step that parses documents into fields before summarising	Figures reach the summary already parsed, rather than re-read from scanned text	High: a new step, a schema and tests
Wrong rules system (27%)	Route to the rules system from the claim type before the tool call	Fewer eligibility checks run against the wrong ruleset	Medium: routing logic and tests
Missed document (15%)	Confirm every attached document is read before summarising	Fewer summaries that omit an attached payslip or letter	Medium: a completeness check

Consolidate. Two candidates target the misread figure. The structured extraction step would also parse every attached document, so it addresses the missed-document failure as well. Record it once, linked to both failure modes, rather than as two separate proposals.

Prioritise and select. The misread figure is the largest share and its prompt-only fix is cheap, so it goes first. The routing check is mid-cost and covers a sizeable share, so it follows. The extraction step covers two failure modes but is costly: defer it unless the cheap prompt change proves insufficient, and keep it linked to both modes it would address.

This leaves a shortlist to validate: the figure-preservation prompt change and the rules-system routing check, with the extraction step held in reserve.

What if a failure mode has no plausible change?

A missing fix does not automatically make the failure acceptable.

Investigate further where the explanation remains unclear, or consider mitigations that reduce the likelihood or consequences of the failure, such as additional checks, a fallback workflow or human review.

If stakeholders agree the remaining failure is acceptable under your requirements, document the decision to accept or defer it, and when to revisit that decision.

Validating improvements

With a shortlist of candidate changes in priority order, the next step is to validate them and ensure they address their corresponding failure mode(s) without breaking anything that matters.

Common validation strategies

Decide how you will work through the shortlist before you start:

Validate independently, then combineEvery change is compared against the same baseline.

Baseline

↓ each change tested on its own

Change A ✓

Change B ✓

Change C ✓

↓ combine

A + B + C ✓one final run validates the combination

Best when

Changes are built in parallel, candidates overlap, or you need each change's own contribution.

Stack in priority orderEach stage adds the next change to the last validated stack.

Baseline

↓ against the baseline

Change A ✓

↓ against the previous stage

A + Change B ✓

↓ against the previous stage

A + B + Change C ✓the final stage is already the full combination

Best when

One team iterates serially, the run budget is tight, or you may stop once acceptance criteria pass.

One at a time

Whichever you choose, two rules always hold: change one factor per comparison, and ship only a combination you have validated as a whole.

Inside each validation run

Under either strategy, every validation run follows the same three steps: decide what to run, decide what to compare against, then read the results in both directions.

Keep the runs themselves comparable as described in Run the System, and expand each step for the practices we recommend:

1. 1Choose the cases to runThe cases that show the failure mode, plus a regression set covering what already worked.
   Recommended practices

   • Cover the failure mode, not just the original examples: include unseen cases that exercise the same behaviour; passing only the cases that prompted the fix proves little.
   • Include a regression set: previously passing cases, core workflows, known edge cases and past bugs.
2. 2Choose the baseline to compare againstAn unchanged reference version, run under identical conditions, changing one factor at a time.
   Recommended practices

   • Match the reference to the decision: the production version for a release decision; the previous candidate or stage to check whether an iteration helped.
   • Hold everything else fixed: the same cases, criteria and evaluator configuration; if you correct the evaluator, re-score both versions.
   • Attribution needs isolation: a multi-change comparison shows only the combined effect, never each change's contribution.
3. 3Check it helped, and that nothing brokeConsider these together: a change is validated only when both hold.3aDid the intended problem improve?Measure the targeted failure directly, and the end-to-end outcomes that depend on it.3bDid anything else regress?Check every product requirement, not just the target; a gain does not offset a failed requirement.
   Recommended practices

   • Label failures consistently: apply the same rules to baseline and candidate outputs, counting every instance of the targeted failure mode, not just the first in each trace.
   • Look beyond the aggregate: check mandatory criteria and important slices, such as capabilities, safety, reliability, latency and cost; a permitted decline is fine.
   • Treat small differences as inconclusive: repeat runs where outputs vary, allow for sample-size uncertainty, and spot-check surprising or newly failing cases by hand.

Worked example: Validating the appeals assistant’s change

Run the revised summarisation instructions through the full workflow. Check whether income figures are preserved accurately and whether recommendation correctness improves in cases where precision matters.

Compare the baseline and candidate on the affected cases, additional document examples and previously passing cases. Then assess the other applicable criteria, such as latency and summary readability. Even if numeric preservation improves, the change is not a validated improvement if a mandatory requirement is no longer met.

Inspect representative outputs and traces to confirm that the measured improvement reflects better behaviour. If results vary or the difference is small, gather further evidence before concluding that the change helped.

An effect is only conclusive once the uncertainty around it is small enough, and the number of samples you evaluate controls that uncertainty.

Worked example: How many samples?

Suppose the requirement is a defect rate below 5%. On 200 samples you observe 3% defects, with a 95% confidence interval of roughly 3% ± 2.4% (0.6% to 5.4%). The upper bound exceeds 5%, so you cannot yet conclude the requirement is met.

Double to 400 samples and the interval narrows to 3% ± 1.7%, an upper bound of 4.7%, now below the requirement. Standard error falls with the square root of the sample size, so halving the margin of error takes four times the samples: there are diminishing returns to adding more.

---

Before you head off

You have now run one pass of the full evaluation loop: defined what "good" means, measured against it, designed a trustworthy evaluation set, executed the system, and diagnosed and improved its failures. From here:

• Go round again. Visit any stage of the loop to refine your definition, metrics, evaluation set, or system. Each pass should increase your confidence in both the system and the evaluation itself.
• Take it live. The next chapter, Productionalising & Scaling the Evaluation, covers deploying and automating the evaluation process.