ORF framework overview

The Observability Requirements Framework is a risk-tiered telemetry specification for generative and agentic AI systems. It connects governance and technical controls to the operational evidence needed to show how a system behaved over time.

Framework structure

System risk tier
      ×
Telemetry priority tier
      ↓
Implementation obligation
      ↓
Selected metrics, logs, and traces
      ↓
Operational and assurance evidence

ORF defines 61 items:

Priority	Items	Purpose
Tier 1	11	foundational interaction, tool, misuse, error, and system records
Tier 2	31	investigation, human oversight, reliability, security, privacy, identity, and change evidence
Tier 3	13	deeper drift, UX, resilience, cost, safety, and governance analysis
Tier 4	6	research-grade or evaluation-intensive outcome, hallucination, groundedness, fairness, and change comparison

Six operational categories

Code	Category	Risks addressed
AGT	Agent-Level Observability	incomplete reconstruction, runaway behaviour, silent regression
PERF	Agent Performance	performance drift, hallucination/ungroundedness, bias/fairness
UX	User Interaction & Behaviour	malicious use, untracked handover/override, experience gaps
SYS	System Performance, Health & Reliability	outage/partial failure, saturation, traffic spike, cost
SEC	Safety & Security Compliance	harmful output, privacy/classified leakage, privilege misuse, mutable audit trail
GOV	Product Lifecycle & Governance	uncontrolled change, ad hoc audit, weak post-incident learning

Three signal types

• Log — a discrete structured record, such as a refusal, error, access, credential, or change event.
• Trace — connected spans/events showing causal or execution relationships across a query, task, retrieval, tool, or service chain.
• Metric — a numerical measure over an observation set or time window, such as rate, count, distribution, coverage, drift, or cost.

Many useful implementations derive metrics from canonical log or trace events. Keep a single source of truth so rates can be reconciled.

Relationship to controls

ORF evidence does not itself prevent harm. It supports controls by showing whether they were invoked and whether they worked.

Risk: runaway tool loop
  → Control: step limit + circuit breaker
  → ORF evidence: tool count, duration, retry lineage, terminal state
  → Use: real-time interruption, incident investigation, control assurance

The framework includes an ARC crosswalk. Some mappings are explicit in the ORF source, while others are thematic interpretations; retain that distinction when using ORF as control evidence.

Key interpretation rules

1. System risk and telemetry priority are different dimensions.
2. The matrix determines obligation; the phase chapters only recommend sequence.
3. Data classification may require stricter telemetry handling regardless of system risk.
4. A waiver accepts residual risk; it does not make the risk disappear.
5. ORF is a baseline and may need system-specific additions.
6. Framework metric definitions should remain stable across implementations.

Continue to Risk and requirements matrix or browse the requirement catalogue.