Implementation overview

Implement ORF as a sequence of usable capabilities. Each phase should leave the service more operable than before; do not postpone dashboards, tests, and ownership until all telemetry is collected.

The three delivery phases

Phase	ORF items	Capability produced
Tier 1 — Foundations	11	reconstruct core interactions, tool use, errors, misuse, and system failures
Tier 2 — Investigation	31	trace across retrieval/services, measure reliability, capture human oversight, and investigate security/privacy events
Tiers 3 and 4 — Analytics	19	evaluate drift, groundedness, fairness, failover, cost, governance, and pre/post-change performance

The phase is a delivery sequence, not the obligation. A High-Risk system must implement Tier 1 and Tier 2, and Tier 3 is mandatory but waivable. A Medium-Risk system must implement Tier 1, while Tier 2 is mandatory but waivable.

Work in vertical slices

Instrument one material journey end to end:

1. create correlation context at the entry point;
2. propagate it through model, retrieval, policy, tool, and human-review operations;
3. emit approved events and metrics;
4. route records through redaction and collection controls;
5. build the investigation view and minimum alerts; and
6. execute an acceptance scenario and retain evidence.

Then expand coverage to other journeys. This exposes schema, access, and operational issues earlier than implementing all producers before testing consumption.

Definition of verified

An ORF item is Implemented when the required signal exists but acceptance or operating verification is still incomplete. Mark it Verified only when:

• the expected event or metric is emitted for all in-scope paths;
• identifiers connect it to related evidence;
• sensitive values are handled according to the approved design;
• pipeline health and loss are observable;
• an authorised user can retrieve or view it within the required time;
• a normal and failure-path test passes; and
• an accountable owner accepts its ongoing operation.

Engineering workstreams

Workstream	Main responsibilities
Instrumentation	events, spans, metrics, schema, correlation, version metadata
Telemetry platform	collection, buffering, routing, storage, indexing, integrity, deletion
Data protection	minimisation, redaction, encryption, access, audit, retention
Operational experience	dashboards, alerts, saved investigations, runbooks
Evaluation	labels, datasets, baselines, sampling, metric calculation
Assurance	requirements status, evidence, waivers, review, residual risk

Minimum non-functional requirements

Set explicit targets for:

• trace and event coverage;
• telemetry ingestion latency;
• tolerated collector or exporter loss;
• evidence query time during an incident;
• access approval and break-glass response;
• retention and deletion completion; and
• telemetry cost and volume budgets.

Phase gates

Gate	Exit question
Design ready	Are schema, content handling, correlation, ownership, and acceptance tests approved?
Foundation ready	Can the team reconstruct normal, failed, refused, and abusive interactions?
Investigation ready	Can the team follow retrieval, services, human decisions, identities, and incident evidence?
Analytics ready	Are metrics based on governed samples, baselines, versions, and response thresholds?
Production ready	Have representative failures been detected, investigated, and handled through runbooks?

Next: Reference architecture