Skip to content

Analytics — Tiers 3 and 4

Tier 3 contains 13 items for deeper performance, safety, experience, resilience, cost, and governance analysis. It is mandatory but waivable for High-Risk systems. Tier 4 contains 6 good-to-have items that usually require labelled or representative evaluation data.

Tier 3 requirements

ORF item Minimum analytical contract
ORF-PERF-01 Define critical-task performance, baseline, current window, difference function, segments, sample size, and response threshold.
ORF-PERF-06 For grounded RAG flows, version top-K recall/retrieval accuracy/embedding-drift method and test corpus.
ORF-UX-05 Use a controlled handover-reason taxonomy: safety, quality, capability gap, policy, and approved extensions.
ORF-UX-07 Define repeated-query matching and publish repeat count/rate per session without raw prompt labels in metrics.
ORF-UX-08 Define abandonment state and timeout; separate user exit, handover, error, and successful completion.
ORF-SYS-10 Test failover attempts and measure successful service recovery, not only infrastructure switch completion.
ORF-SYS-13 Allocate model, tool, retrieval, compute, and platform cost to a query/task/session with documented assumptions.
ORF-SEC-01 Define harmful-output label source, evaluator, denominator, coverage, and human-review escalation.
ORF-SEC-04 Use approved bounded content categories and monitor distribution shifts, especially in sensitive topics.
ORF-GOV-01 Measure rollout frequency and percentage of changes with recorded safety evaluation.
ORF-GOV-02 Measure audited-session coverage and findings per period, with risk-based sampling.
ORF-GOV-04 Record reviewed session, reviewer, finding taxonomy, severity, action, and closure.
ORF-GOV-05 Measure incidents by severity and recurrence using a documented definition of materially similar cause.

Tier 4 requirements

ORF item Minimum analytical contract
ORF-AGT-14 Compare versioned pre/post-change metrics; report absolute and relative change, sample size, and release decision.
ORF-PERF-03 Label outcomes as safe failure, unsafe failure, unsafe success, or safe success; define task success and safety independently.
ORF-PERF-04 Measure materially incorrect responses on a governed sample with clear reviewer rubric and adjudication.
ORF-PERF-05 Define missing/weak citation for high-risk domains; measure count and rate on the eligible denominator.
ORF-PERF-07 Define reference groups, task performance, parity measure, sample adequacy, and approved use of sensitive attributes.
ORF-PERF-08 Compare hallucination/refusal rates across approved segments and report uncertainty and sample size.

Build an evaluation protocol

Every analytical metric needs:

  • a decision it informs;
  • population and sampling method;
  • dataset and label provenance;
  • evaluator type, rubric, and version;
  • metric formula and exclusions;
  • minimum sample size and uncertainty treatment;
  • relevant segments and privacy approval;
  • baseline and change threshold;
  • calculation cadence; and
  • owner and response action.

Avoid common analytical failures

  • Evaluator drift: changing an LLM judge or rubric breaks the time series unless results are recalibrated.
  • Selection bias: feedback and incident samples do not represent all interactions.
  • Small denominators: segment rates can look extreme with very few examples.
  • Metric gaming: completion rate alone can reward unsafe success; refusal rate alone can reward over-refusal.
  • Version mixing: aggregate results can hide that one prompt, model, or retrieval version regressed.
  • False precision: noisy labels should include disagreement, confidence, or uncertainty where relevant.

For material changes, compare safety, quality, latency, cost, refusal, handover, and incident-leading indicators on the same representative task set. Define rollback thresholds before rollout and monitor canary cohorts separately.

Phase exercise

Run an evaluation using two system versions. Demonstrate reproducibility from dataset and configuration references, explain every metric denominator, inspect the largest segment difference, and show the release decision and follow-up action.

Exit criteria

  • all required Tier 3 items are Verified or validly waived;
  • Tier 4 items selected for material system risks have governed protocols;
  • metrics are reproducible and versioned;
  • raw samples and sensitive attributes have approved access and retention;
  • thresholds lead to named actions; and
  • operators and governance reviewers understand limitations, not only point values.

Next: Implement by operational category