Runbooks and response
Runbooks turn ORF evidence into safe, repeatable action. Use the organisation's incident process and add AI-specific evidence, containment, and recovery decisions.
Minimum runbook set
| Runbook | Key evidence | Important containment decision |
|---|---|---|
| Service degradation/outage | SLOs, component traces, timeouts, queues, release | fail over, degrade safely, or suspend service |
| Runaway/looping task | task tree, step/tool counts, retries, external state | interrupt; revoke tool access; reconcile side effects |
| Harmful/policy output | policy decision, content evidence, version, affected cohort | block flow/release; strengthen review or guardrail |
| Privacy/classified leakage | detection, restricted evidence, access logs, trace | contain exposure; restrict access; invoke notification process |
| Unauthorised tool/credential use | identity, scope, permission/token events, tool result | revoke credential; disable tool; preserve audit evidence |
| Quality/groundedness regression | evaluation sample, retrieval, version, feedback/overrides | rollback, route to human review, or narrow scope |
| Evidence pipeline failure | collector/export loss, coverage, affected time/journeys | fail closed for critical actions or activate approved fallback |
Standard runbook structure
- Purpose and trigger — condition, severity, and scope.
- Ownership — primary, backup, and escalation route.
- Immediate safety check — ongoing harm, external actions, data exposure, and evidence availability.
- Triage queries — dashboards, saved searches, trace, change, identity, and case links.
- Containment — pause, block, revoke, isolate, fail over, increase human review, or narrow functionality.
- State reconciliation — determine whether external actions or data changes completed.
- Recovery — safe restart, rollback, reprocessing, or user follow-up.
- Communication — approved internal and external channels; exclude sensitive evidence.
- Evidence preservation — case reference, integrity-protected trace, access record.
- Closure and learning — impact, root cause, actions, owner, due date, recurrence link.
Use the runbook template.
The first five questions
During triage answer:
- Is harm ongoing, and can the affected capability be safely stopped?
- Which users, tasks, versions, tools, data, and environments are affected?
- Did any external or irreversible action occur, and is final state known?
- Is the evidence complete and trustworthy for the affected window?
- What changed before the issue began?
Preserve evidence safely
Flag and retain only the approved evidence needed for the incident. Apply case-based access, classification, integrity controls, and audited export. Do not paste raw prompts, personal data, credentials, or classified content into general chat or ticket fields.
Exercise cadence
Before go-live, exercise the most consequential failure and evidence-pipeline failure. Repeat after major architecture/capability changes and at the approved periodic cadence. Rotate participants so the runbook does not depend on its author.
Measure:
- time to detect;
- time to identify affected version/journey;
- time to retrieve evidence;
- time to contain;
- percentage of state reconciled; and
- missing or ambiguous evidence found during the exercise.
Post-incident learning
Connect root cause and corrective actions to ORF requirement IDs. Ask whether the control failed, the signal failed, the alert failed, the response failed, or the risk was not modelled. Update the requirements register, telemetry design, test suite, and risk decision where needed.
Done when: an operator unfamiliar with the implementation can use the runbook and authorised evidence to reach a safe containment decision within the target time.
Next: Reviews and governance