Skip to content

Getting started with ORF

The Observability Requirements Framework (ORF) helps a team decide what evidence an AI system must produce, how that evidence should be connected, and how it will be used to operate and govern the service.

Implementation guidance

This playbook supports implementation and internal review. Completing its activities or templates does not by itself establish compliance, completeness, or approval. Decisions remain subject to the organisation's authorised risk, security, privacy, architecture, operational, and governance processes.

ORF covers 61 telemetry items across six operational categories. The requirements include metrics, logs, and traces for model interactions, tool use, user behaviour, system reliability, safety, security, and governance.

What success looks like

An ORF implementation is successful when the team can:

  1. reconstruct a material user session from entry point to final response or action;
  2. detect important reliability, safety, security, and quality failures;
  3. investigate an incident without assembling evidence manually from unrelated systems;
  4. demonstrate which requirements apply, which have been implemented, and which have an approved waiver; and
  5. use telemetry findings to improve the service and its controls.

What ORF is — and is not

ORF is ORF is not
A risk-tiered telemetry specification A replacement for an AI risk assessment
An evidence layer for technical and governance controls A mandate to collect every possible data field
Technology-agnostic A requirement to buy a particular observability product
Applicable throughout the system lifecycle A one-off go-live checklist
Proportionate to system risk Permission to store prompts or personal data without safeguards

Observability can create a second data-risk surface

Prompts, responses, retrieval content, memory, and tool parameters can contain personal, classified, or otherwise sensitive information. Treat telemetry as production data: minimise it, classify it, restrict access, define retention, and test deletion.

The implementation journey

Stage Key question Main output
1. Understand What does the system do, for whom, and across which components? System profile and boundary diagram
2. Scope What is the system risk tier and which ORF requirements apply? Risk decision and requirements register
3. Design What will be emitted, linked, protected, retained, and used? Telemetry design
4. Implement Can the platform produce the selected logs, traces, and metrics? Instrumented service and operational views
5. Verify Can the team detect and investigate representative failures? Test evidence and implementation-readiness record
6. Operate Are signals reviewed and controls improved over time? Runbooks, review records, and improvement backlog

Suggested working model

Use a cross-functional working session for planning. Include the product owner, application or AI engineer, platform/SRE representative, operations lead, and the relevant security, privacy, and governance representatives. A small system may combine roles, but every responsibility still needs an owner.

For a first implementation, work through one representative user journey end to end before trying to cover every use case. Choose a journey that includes the system's most consequential model decision, retrieval, or tool action.

Before the first workshop

Bring:

  • the system description and intended-use statement;
  • a current architecture or data-flow diagram;
  • the AI risk assessment and data classification, if available;
  • the list of models, prompts, retrieval sources, tools, and external services;
  • two or three representative user journeys and failure scenarios; and
  • existing monitoring, incident-response, and change-management standards.

Outputs to retain

Next: Is ORF for my system?