Skip to content

Operational runbook worksheet

Create one runbook for each material alert or incident scenario. A runbook is complete only after an operator who did not write it can use the linked evidence to make the intended decision.

Example scenario used below: obsolete retrieval content causes incorrect public guidance.

1. Document control

Field Record
Runbook / scenario Enter: signal and failure scenario
Service and capability Enter: service, environment, and affected journey/tool
Primary / backup owner Enter: on-call role and escalation backup
Severity and response target Enter: severity criteria, acknowledgement, containment, and recovery targets
Last tested / next test Enter: exercise dates and result link
Related ORF IDs Enter: signals used, such as ORF-AGT-03, ORF-AGT-04, ORF-GOV-03

2. Trigger and scope

Question Example Your runbook
What triggers this runbook? Citation weakness above threshold after a retrieval-index release Enter: alert, report, or manual condition
Which query confirms it? Compare weak-citation rate and source IDs by index version Link: dashboard or saved query
What are known false positives? Approved policy documents without stable paragraph anchors Enter: conditions and validation method
What may be affected? Public guidance journey, retrieval service, model answers, officer queue List: users, journeys, environments, dependencies, and external state

3. Immediate safety check

  • Is harm, data exposure, or unauthorised action ongoing?
  • Can the affected capability be stopped or degraded safely?
  • Could an external or irreversible action have occurred?
  • Is the final external state known and reconciled?
  • Is telemetry complete, timely, and trustworthy for the incident window?
  • Is restricted evidence access required and authorised?

Record each answer in the incident timeline; do not rely on checklist state alone.

4. Triage queries

Question Evidence to use Expected interpretation or next action
Which release is affected? Change log, prompt/model/index versions, deployment marker Isolate the first bad version and previous known-good version
Which journeys and users are affected? Trace/response IDs grouped by source and release Bound impact without exposing content in general channels
What changed before onset? Approved change and evaluation records Confirm owner, approver, scope, and rollback path
Which component or control failed? Retrieval spans, policy outcomes, errors, and coverage health Distinguish bad source/ranking from model or control failure
Is evidence trustworthy? Trace coverage, schema validity, ingestion delay, redaction failures Mark conclusions provisional if the evidence pipeline is impaired

5. Containment options

Option Authority Side effect Verification
Disable affected topic/capability Enter: authorised role Users lose automated guidance Confirm safe handover/fallback and no new affected traces
Roll back index/prompt/model Enter: release authority Reverts other intended changes Confirm version marker and recovery evaluation
Revoke tool or credential access Enter: security/operations role Transactions may stop or remain incomplete Reconcile external state and credential-use logs
Increase human review Enter: service owner Queue load and response time increase Confirm queue acceptance, capacity, and reviewer guidance
Degrade to safe fallback Enter: incident commander Reduced capability Test fallback journey and user messaging

Pre-authorise low-risk containment actions where possible. Record who made each decision and why.

6. Reconciliation and recovery

  1. Identify external transactions or communications whose final state is unknown.
  2. Deduplicate by stable request/idempotency reference.
  3. Apply approved compensating actions; never replay blindly after a timeout.
  4. Restore the known-good configuration or complete the corrective change.
  5. Run normal, failure, privacy, and telemetry-pipeline acceptance tests.
  6. Confirm dashboards and alerts represent the restored version.
  7. Obtain the required restoration approval and monitor the defined recovery window.
  8. Notify or remediate affected users where required.

7. Escalation and communication

Condition Notify and channel Decision authority Evidence handling
Ongoing harm, data exposure, or unauthorised action Incident, security/privacy, product, and governance channels Incident commander plus relevant control authority Restricted case reference only in general channels
Consequential external state unknown Integration owner and service owner Named reconciliation authority Tool request IDs and restricted evidence link
Evidence pipeline untrustworthy Platform/SRE and affected metric owners Platform incident lead Mark dashboards stale and suspend evidence-dependent automation
Enter: system-specific threshold Enter: role and approved channel Enter: named authority Enter: classification and sharing rule

8. Evidence and closure

  • Incident/case ID and protected evidence reference: enter links.
  • Timeline and decisions: include actor, timestamp, evidence, and rationale.
  • Impact and root cause: distinguish confirmed facts from hypotheses.
  • Corrective actions: record owners, dates, and verification criteria.
  • Recurrence link: connect to earlier root cause or failed action where relevant.
  • Artefact updates: update the system profile, requirements register, telemetry design, tests, and runbook.
  • Closure: record approver, date, residual risk, and monitoring period.

Next: retain the exercise or incident result in the evidence register.