Skip to content

Worked metric examples

These examples show the complete calculation and the operating interpretation. Values are fictional.

1. Trace coverage — ORF-AGT-04

During one hour, the journey definition expects 9,720 traceable events. Of these, 9,526 contain the expected trace ID.

trace coverage = events with trace ID / expected traceable events
               = 9,526 / 9,720
               = 98.00%

If the target is 99.5%, this is a failure even when service availability is healthy. Investigate the missing producer, asynchronous boundary, or collector loss before trusting incident and rate metrics for the window.

Report: 9,526 / 9,720 = 98.00%; target 99.5%; Fail.

2. Retry rate — ORF-AGT-11

There are 480 logical tool calls. Twenty-four calls retry once and six of those retry a second time, producing 30 retry attempts.

retry attempts = total attempts − logical calls
               = 510 − 480
               = 30

retry rate = retry attempts / logical calls
           = 30 / 480
           = 6.25%

calls affected by retry = 24 / 480 = 5.00%

Both measures are useful: 6.25% describes extra load; 5.00% describes user/tool calls affected. Deduplicate replayed telemetry before calculation.

3. Availability — ORF-SYS-01

A 30-day month contains 43,200 minutes. The user-facing service is unavailable for 63 minutes.

availability = (total time − downtime) / total time
             = (43,200 − 63) / 43,200
             = 99.854%

Against a 99.9% target, the monthly downtime budget is 43.2 minutes, so the service exceeds it by 19.8 minutes.

Define downtime from the user journey

A healthy web process is not available if the model, retrieval, tool, or human queue cannot produce the promised safe outcome.

4. Refused share — ORF-SEC-02

The policy layer detects 320 violation attempts. It refuses 278, redacts and safely answers 22, escalates 14 to a person, and records 6 control errors.

refused share = refused violations / detected violations
              = 278 / 320
              = 86.88%

Do not interpret the remaining 13.12% as automatically unsafe. Publish the full action distribution:

Outcome Count Share
Refused 278 86.88%
Redacted/answered 22 6.88%
Escalated 14 4.38%
Control error 6 1.88%

Displayed shares total 100.02% because each category is rounded independently.

The six control errors require investigation even though the headline refusal rate looks high.

5. Hallucination rate — ORF-PERF-04

A governed random sample contains 400 eligible responses. Reviewers adjudicate 17 as materially incorrect.

hallucination rate = materially incorrect responses / sampled responses
                   = 17 / 400
                   = 4.25%

Report the sample, period, task mix, system versions, reviewer rubric, adjudication method, and uncertainty. An approximate 95% Wilson interval is 2.7%–6.7%, showing why 4.25% should not be presented as exact population truth.

Compare only with results produced under a compatible sampling and review protocol.

6. Performance drift — ORF-PERF-01

Critical-task accuracy is 88% at baseline and 82% in the current governed sample.

absolute change = 82% − 88% = −6 percentage points
relative change = (82% − 88%) / 88% = −6.82%

Say “down 6 percentage points” for the absolute change, not “down 6%.” Investigate task mix, data/source changes, model/prompt versions, and evaluation-method changes before attributing the drift.

7. Cost per task — ORF-SYS-13

For one day:

Cost component Amount
Model/token charges $615
Retrieval and tool charges $95
Allocated platform cost $120
Total $830

There are 18,500 completed or safely failed tasks.

cost per task = total allocated cost / eligible tasks
              = $830 / 18,500
              = $0.0449 per task

Document allocation assumptions and publish task outcomes alongside cost. A lower unit cost caused by premature refusal or degraded retrieval is not necessarily an improvement.

8. Incident recurrence — ORF-GOV-05

There are 12 incidents in a quarter. Three materially repeat a previously identified root cause or failed corrective action.

recurrence rate = recurring incidents / incidents
                = 3 / 12
                = 25%

Use a reviewed root-cause link rather than a broad incident category. The action is to examine overdue or ineffective corrective actions, not merely to reduce the label count.

Metric review checklist

For every metric, show:

  • count and denominator;
  • observation window and time zone;
  • source-event and coverage status;
  • formula/calculation version;
  • included and excluded populations;
  • relevant system/model/prompt/data versions;
  • sample size and uncertainty where applicable;
  • target/threshold and accountable owner; and
  • action when the result crosses the threshold.