Is ORF for my system?
Use ORF for a generative or agentic AI system when operating it safely depends on evidence of what the model, orchestration layer, retrieval system, tools, users, or operators did.
Quick applicability check
ORF is likely applicable if the system does one or more of the following:
- generates answers, summaries, recommendations, decisions, or content;
- retrieves information from a knowledge base, memory, search service, or external source;
- invokes tools, APIs, workflows, code, or other agents;
- influences a consequential public or internal process;
- processes personal, confidential, or classified data;
- uses human review, override, escalation, or handover;
- must support audit, incident investigation, or post-deployment evaluation; or
- may cause cost, reliability, safety, security, fairness, or reputational harm when it fails.
If none apply, conventional application and infrastructure observability may be sufficient. Record that decision and revisit it if the system's capabilities or use change.
Choose a proportionate starting point
| Situation | ORF starting point |
|---|---|
| Internal experiment; outputs do not inform downstream decisions | Profile the system and adopt useful Tier 1 telemetry. |
| Internal assistant; a person reviews every output | Complete planning and implement Tier 1. Add applicable Tier 2 items for investigation and handover. |
| Public information service | Treat user abuse, refusal, privacy, groundedness, and service reliability as first-class concerns. |
| Agent that performs external actions | Trace the complete decision–tool chain, credential use, retries, partial failures, and human approvals. |
| Consequential or high-impact service | Apply the High-Risk matrix and establish formal waiver, review, and incident-evidence processes. |
Cases requiring special care
Retrieval-augmented generation
Capture which sources were queried and retrieved, their version or identifier, and how retrieval connects to the response. Avoid copying full documents into telemetry when stable references and authorised retrieval are sufficient.
Tool-using or action-taking agents
Capture the task, selected tool, parameters after safe redaction, authorisation context, result, retry lineage, and final state. A successful tool call and a successful user outcome are not the same thing.
Multi-agent systems
Propagate trace context across delegated tasks. Record the parent task, receiving agent, delegated objective, result, and control decision so accountability is not lost at agent boundaries.
Human-in-the-loop systems
Record when review was requested, what the reviewer could see, the decision, reason, timestamp, and resulting system action. Do not interpret the presence of a human as proof that effective oversight occurred.
Decision record
Record:
- whether ORF applies;
- the systems and environments in scope;
- the rationale and decision owner;
- known exclusions and compensating controls; and
- the next review trigger or date.
Done when: the product, technical, and risk owners agree whether ORF applies and no material system component is implicitly outside the boundary.
Next: How to use this playbook