Assessing Performance
To assess system performance, we first need to answer the question your acceptance criteria pose:
Which criteria does the system meet, where does it fall short, and by how much?
Currently, each test case already has a score on each of your metrics, such as pass/fail on correctness or a 0–1 faithfulness score.
Yes, but that aggregation only completes the metric's definition, turning judgements into a score for each case.
A score per case is not yet a verdict. To gain a more comprehensive assessment, here are two levels of insight you can extract from the results:
- The baseline check (the bare minimum). For each acceptance criterion, summarise its cases into one figure and compare that figure to the criterion's threshold. This per-criterion check is the verdict you must produce.
- Deeper insight (optional). Split the results, by group of cases, by family of similar criteria, or across versions, to see where a shortfall concentrates and to surface failure modes the criteria did not anticipate.
This page covers the baseline first, then the two mechanics behind every figure in it: splitting (which sets of cases to look at) and summarising (how a set becomes one figure).
Baseline check: Assessing against acceptance criteria
Start by determining whether your system meets the acceptance criteria established earlier. For each criterion, calculate the relevant metric across its applicable cases and compare it against predefined threshold(s).
This provides an initial verdict on whether your system meets its requirements. The analyses that follow go beyond this verdict, uncovering performance gaps, unexpected patterns, and potential failure modes that may warrant further investigation.
Deeper insight: Exploring the results
Splitting results
An overall figure can hide important differences. Split the results into possibly meaningful groups of cases to yield further insights on where the system thrives or struggles.
Group along dimensions tied to your requirements, and, if possible, along ones you did not originally plan for but might still bear on your objectives. For example:
- "Can GrantsAssist handle different types of appeals accurately?": Splitting by appeal type aligns with the initial requirement to handle previously defined appeal types, and may reveal a failure mode in one of them.
- "Does the complexity of an appeal affect response quality?": Splitting by appeal length or number of issues raised might help investigate potential weaknesses not anticipated during evaluation design.
If you repeated runs with the same configurations to manage reliability, each case has multiple outputs for each criterion. Aggregate each case's runs first, before any further splits; every split and summary below assumes one result per case.
Common ways to split results
Within a single run, here are two common axes that you can consider to split the results to yield richer insights:
- By group of cases. Slice the cases into meaningful groups, such as appeal type, difficulty, length, or user segment.
- By criterion. Read each acceptance criterion on its own, or group similar criteria into a family where aggregating is meaningful, such as correctness and faithfulness as an "accuracy" family.
The grid below lets you explore both: rows are groups of cases, columns are criteria, and each cell is that group's score on that criterion.
Showing cases split into groups, each criterion on its own. Figures are v2, with the change from v1 in each cell. Pooled figures take a simple mean of their parts; how much each case should count is covered under Summarising each split below.
Splitting along multiple dimensions can provide a more comprehensive view of the results, if done correctly. Crossing dimensions works well: version × group shows whether v2's gains hold for appeals, and criterion × group shows which criteria fall short in which groups. Pooling or averaging a dimension away is counterproductive: pooling across versions gives one rate that describes neither v1 nor v2, and averaging criteria blends correctness and tone into a single number.
Summarising each split
Whether or not you split, each set of cases should be summarised into one single figure for easy interpretation.
There is no single right way to summarise results, but there are wrong ones: an unsuitable summary can distort the results and lead to inaccurate conclusions.
How much each case counts
When the sets differ in size, you can vary how much each group counts. Explore the ways to weight them below:
Pool all cases. Each case counts equally, so large groups dominate.
| Group | Cases | Pass rate | Weight |
|---|---|---|---|
| Eligibility | 800 | 90% | 80% |
| Application help | 150 | 80% | 15% |
| Appeals | 50 | 40% | 5% |
Micro-average answers: How often does a case from this set pass?
Suits: Comparing products on the same set
Default to micro-averaging when your evaluation set already reflects the mix of cases you care about.
Some cases may not run, and some judgements may be invalid. Re-run the cases where possible, holding everything else constant. If not, fall back on one of these:
- Exclude it from the denominator when the gap says nothing about the system, such as an invalid judgement.
- Impute a value in its place:
- As a fail: when users would see the failure, such as a timeout or crash.
- As the mean: rarely appropriate; it assumes the gap would have scored like the rest.
Worked example: Recording results so gaps stay visible
To split or treat results later, each result must carry its labels. Record one row per test case, run, and criterion, with a column for every split your objectives need. If a label is missing, such as which group a case belongs to, you cannot split by it afterwards.
In the example below, Q-0142's runs are combined within the test case as described above. Q-0143 timed out, so it counts as a fail. Q-0587's correctness judgement was unparsable, so it is re-judged or excluded, while its tone judgement still counts:
| Case | Run | Group | System | Criterion | Score | Status |
|---|---|---|---|---|---|---|
| Q-0142 | 1 | Appeals | v2 | Correctness | Pass | Valid |
| Q-0142 | 2 | Appeals | v2 | Correctness | Fail | Valid |
| Q-0143 | 1 | Appeals | v2 | Correctness | N/A | System timeout |
| Q-0587 | 1 | Eligibility | v2 | Correctness | N/A | Invalid judgement |
| Q-0587 | 1 | Eligibility | v2 | Tone | 4 / 5 | Valid |
Keep failed and invalid rows rather than deleting them. The status column is what lets you apply the treatments above.
Was this page helpful?