Analysing Failures
Assessing performance tells you where the system falls short of your acceptance criteria. This page examines the cases behind those gaps to understand what went wrong and which failure patterns recur. The aim is a failure taxonomy, with a set of causal hypotheses, that is:
- Comprehensive: Every failed acceptance criterion can be traced to at least one cause.
- Lean: The causes are as distinct and few as possible.
The process is an iterative loop:
The richer the traces you captured during execution (intermediate outputs, tool calls, retrieved context, reasoning steps), the more you can see inside each failure. Shallow outputs tell you that a case failed; deep traces tell you where in the pipeline it went wrong.
Open coding: reading and labelling traces
Open coding means reading traces one at a time and noting whatever feels wrong, with no predefined categories. The goal is to let failure patterns surface from the data rather than imposing structure on it.
Open coding depends on careful judgement and an understanding of what counts as a failure in your context, so we do not recommend delegating it to an LLM.
However, for long or complex traces, an LLM can still help you read rather than judge: highlighting anomalies, summarising a step, or pointing to where in the pipeline to look. Keep the verdict and the failure notes your own.
Here are some tips for reading traces and taking notes:
- Note what feels wrong. Jot a brief note wherever an output is incorrect, an action surprising, or behaviour off. Each note is a candidate failure to group later.
- Favour speed. Keep notes short; full sentences are not needed.
- Read as the user. Judge from their perspective, not the system's.
- Note the first failure, not every one. Fixing the first error often clears the rest of the chain.
- Give a holistic verdict. Mark each trace acceptable or unacceptable; a firm yes or no sharpens thinking more than a vague score, especially on borderline cases.
- Stuck? Use known failure modes as lenses. Scan for common modes such as hallucination, or application-specific ones such as malformed queries in a text-to-SQL system or policy-breaking replies in a customer-service bot. Use them to spot problems, not to force traces into categories.
- Know when to stop. Stop when new failure modes stop appearing; let saturation, not a fixed count, be the signal.
A Likert scale asks annotators to rate a quality such as relevance or helpfulness on a numeric scale, typically 1 to 5. These scales are widely used, but without a detailed rubric they add substantial noise: annotators agree less and subjective variance rises. A binary decision about a specific failure mode (did the problem occur or not) produces far more reproducible annotations.
Worked example: Open coding the appeals assistant's traces
Reading through the failing traces, you jot a short note on the first thing that feels wrong in each, plus an acceptable or unacceptable verdict:
| Trace | Note | Verdict |
|---|---|---|
| #4 | Income read as $2,400, but the payslip says $4,200; digits transposed | Unacceptable |
| #7 | Queried the residency rules system; this case turns on income rules | Unacceptable |
| #9 | Summary omits the second payslip the applicant attached | Unacceptable |
| #12 | Recommendation is correct, but the rationale cites the wrong clause number | Unacceptable |
| #15 | Recommends decline; income is actually under the limit once read correctly | Unacceptable |
| #18 | Correct recommendation and rationale; phrasing is a little verbose | Acceptable |
The notes are raw and ungrouped, and the verdicts commit to a side even when a case feels borderline (trace #18). Grouping these observations into categories is the next step, axial coding.
Axial coding: categorising failures
Axial coding turns those freeform notes into a structured set of failure categories, a taxonomy. Group similar observations together, name each group, and count how often it occurs. This is where pattern analysis happens: a category that covers many failures is a far stronger lead than any single case.
Here are some common groups to consider when clustering failures:
Group by cause, not surface similarity. Two failures that share a label may still be different categories: split them when their causes and fixes differ, and merge them when they would lead to the same fix.
For instance, a property assistant's "hallucination" failures split into two:
| Category | Cause | Example |
|---|---|---|
| Hallucinated listing details | Invents details absent from the listing data | Claims a home has solar panels when it does not |
| Hallucinated user actions | Misreads or fabricates user intent | Schedules a tour the user never requested |
This is only a first pass; expect to split or merge categories as your understanding of the failure modes deepens.
LLMs lean towards generic failure categories, such as hallucination or tone, rather than the application-specific ones that only become visible by reading real traces. A taxonomy built that way may not match your actual failure distribution.
Once your categories exist, an LLM can help with clustering and organising new observations into them.
Worked example: Axial coding the appeals assistant's notes
Grouping the open-coded notes into distinct failure modes, and merging symptoms that trace to the same one, gives a lean taxonomy:
| Failure category | Stage | Share of failures |
|---|---|---|
| Misread a numeric figure | Summary | 48% |
| Called the wrong rules system | Tool-call | 27% |
| Missed a document entirely | Summary | 15% |
| Other | Mixed | 10% |
Each row is a distinct failure mode that will need a distinct fix. A downstream symptom such as a wrong recommendation is not its own category: it is merged into whichever failure produced it (a misread figure, or the wrong rules system).
Testing explanations (optional)
Skip this step when you are already confident about a failure category's cause, particularly for the dominant categories. It is also a question of resources: testing explanations takes time, and is rarely worth it for minor failure modes.
By now you have a set of failure categories, some with an obvious cause and others still uncertain. Testing an explanation confirms which cause is really behind a category, so the fix you design later targets the right thing.
For each uncertain category, sketch a hypothesis or two and then test them. We recommend starting with these two approaches:
The first thing you can try.
- Re-run all affected cases with more in-depth traces; inspect what the uncertain step received and produced.
- Identify where the failure mode first appears and rule out hypotheses that cannot explain it.
- Annotate these traces again and check whether the failure mode is still consistent for all affected cases.
When the trace alone cannot separate two explanations.
- Make one targeted change per hypothesis, and state the result you expect.
- Compare against the unchanged system: vary one thing at a time, keep the test set fixed.
- Repeat if runs vary. Check that any difference is attributable to the change, not random variation.
Worked example: Testing why figures are misread
The dominant category, misread numeric figures at the summary stage, has two plausible causes:
- The model reads figures from scanned tables unreliably.
- The model reads them correctly but rounds during summarising, losing precision that matters near an eligibility threshold.
Reproduce first. Re-run the affected cases with more in-depth traces and inspect the summary step. A figure that is already wrong there places the error before the recommendation, though it does not yet tell misreading apart from rounding. Re-annotate the traces to confirm the failure is consistent across the affected cases, and keep them for re-testing after a change.
Then compare. When the trace cannot separate the two, give each hypothesis a targeted test, an ablation or a controlled change, with a stated expectation, run against the unchanged system:
| Hypothesis | Targeted test | Evidence that supports it |
|---|---|---|
| Scanned tables are misread | Replace the scanned input with a verified transcription of the same figures, leaving the summary instructions unchanged | Fewer numeric errors |
| Figures are rounded while summarising | For cases whose input already holds the correct figures, change the instructions to require exact values, with no rounding | Fewer rounding-related discrepancies |
Neither test alone proves the exact mechanism, but each shifts the evidence towards or away from its hypothesis. Repeat a comparison if runs vary, so the difference is attributable to the change rather than random variation.
When to stop
Stop when you achieve a lean, comprehensive failure taxonomy: every failed acceptance criterion can be traced to at least one cause, and the causes are as distinct and few as possible.
Evaluation practitioners such as Hamel Husain suggest that two focused passes of open and axial coding are usually enough to reach a lean, comprehensive failure taxonomy.
With a lean, comprehensive failure taxonomy in hand, continue to designing improvements.
Was this page helpful?