Agent performance (PERF)
Agent performance telemetry detects drift, incorrect or ungrounded responses, ineffective human oversight, and unfair differences across relevant groups. Unlike infrastructure metrics, these measures require an explicit definition of a “good” or “safe” task outcome.
Requirement map
| Tier | ORF items | Capability |
|---|---|---|
| 2 | ORF-PERF-02 |
human override and edit evidence |
| 3 | ORF-PERF-01, ORF-PERF-06 |
critical-task and retrieval drift |
| 4 | ORF-PERF-03–ORF-PERF-05, ORF-PERF-07, ORF-PERF-08 |
outcome safety, hallucination, groundedness, and fairness analysis |
Define task performance first
For each critical task, document:
- expected outcome and unacceptable outcome;
- evaluation unit: response, task, session, or external action;
- automatic and human evidence available;
- target population and important segments;
- baseline, threshold, and minimum sample size; and
- action when the threshold is crossed.
Do not use generic “accuracy” across unrelated tasks. A summarisation assistant, public information chatbot, and action-taking case agent need different performance contracts.
Separate success from safety
Use the ORF outcome model:
| Outcome | Task completed? | Safe? | Example |
|---|---|---|---|
| Safe success | Yes | Yes | correct grounded answer or authorised action |
| Safe failure | No | Yes | appropriate refusal or handover |
| Unsafe failure | No | No | failed task that leaks data or leaves uncertain external state |
| Unsafe success | Yes | No | completed action or answer that violates policy |
This prevents teams from improving completion at the expense of safety, or improving refusal rate by refusing legitimate requests.
Govern hallucination and groundedness review
Define “materially incorrect” and “weak citation” for the domain. Use a representative, versioned sample and record reviewer rubric, reviewer agreement/adjudication, evidence sources, and metric coverage. Where an automated evaluator is used, calibrate it against human-reviewed examples and version the evaluator.
For RAG, measure retrieval separately from generation. A response can be ungrounded because the correct source was not retrieved, because an irrelevant source was ranked highly, or because the model ignored a correct source.
Human override evidence
Record whether an output was accepted, edited, rejected, or escalated, plus a controlled reason category. Separate substantive correction from stylistic editing. A rising edit rate can mean poorer system quality, a changing task mix, or stricter reviewers; investigate before drawing conclusions.
Fairness and segment analysis
Use sensitive attributes only with a lawful, approved purpose and strong access controls. Define reference groups, appropriate parity metric, sample adequacy, and known confounders. Report counts and uncertainty; do not rank or act on very small segments without review.
Operational views
- critical-task performance by version and cohort;
- human edit/override rate and reason;
- retrieval quality and source freshness;
- safe/unsafe × success/failure outcomes;
- hallucination and ungrounded rates with coverage;
- segment comparison with sample size; and
- change annotations and evaluation-method version.
Acceptance scenarios
- Insert known-answer examples and confirm incorrect responses enter the governed review workflow.
- Degrade retrieval ranking and confirm the retrieval metric changes before the aggregate service metric.
- Compare two prompt/model versions using the same task set and evaluator version.
- Recalculate a segment rate from source labels and reconcile it with the dashboard.
Done when: performance measures have decision owners, reproducible inputs, stable definitions, adequate coverage, and defined remediation—not merely attractive charts.
Next: User interaction