Reviews and governance
Review ORF evidence at a cadence matched to service risk and change rate. Reviews should produce decisions and owned actions, not only dashboard walkthroughs.
Suggested review cadence
| Review | Suggested cadence | Focus | Participants |
|---|---|---|---|
| Operational health | weekly or service-defined | SLOs, alerts, pipeline coverage, incidents, queues, costs | product, engineering, SRE/operations |
| Safety and quality | fortnightly/monthly, risk-based | refusals, overrides, evaluations, harmful/ungrounded samples, user patterns | product, AI/evaluation, operations, risk |
| Security and privacy | monthly and after incidents | leakage, privileged tools, credentials, access, audit integrity | security, privacy, technical owner |
| Change/release | each material change | evaluation, approvals, rollout thresholds, canary, rollback | change owner, technical, product, risk as required |
| ORF assurance | quarterly or risk-based | requirement status, waivers, evidence tests, residual risk | accountable owner, risk/governance, control owners |
| Scope/risk reassessment | annually and on trigger | intended use, users, autonomy, tools, data, impact | product, technical, security/privacy, governance |
Adapt frequency to agency policy. A rapidly changing High-Risk system may require more frequent review.
Standard agenda
- material service, user, model, prompt, policy, data, tool, or architecture changes;
- telemetry health and evidence gaps;
- incidents, near misses, recurrence, and overdue actions;
- reliability, safety, security, experience, and performance trends;
- changed baselines, thresholds, datasets, or evaluators;
- waivers approaching expiry and unmet mandatory requirements;
- access, retention, deletion, and audit-trail concerns; and
- decisions, owners, due dates, and reassessment needs.
Review evidence quality
Ask:
- Is the denominator complete and is collection coverage known?
- Did a version, taxonomy, sampling, or evaluator change break comparability?
- Are aggregates hiding a critical journey or segment?
- Are alerts actionable and runbooks tested?
- Can findings and incidents be linked to corrective changes?
- Are content collection and retention still proportionate?
Waiver review
At expiry, reassess the underlying risk, implementation feasibility, compensating controls, incidents, and system changes. Do not automatically renew. If the system risk increases, an item may move from waivable to mandatory.
Record decisions
Retain meeting date, evidence reviewed, attendees/roles, decisions, challenged assumptions, accepted residual risk, actions, owners, due dates, and follow-up status. Link the review record to the evidence register rather than copying sensitive evidence into minutes.
Done when: recurring reviews identify changes and gaps, make explicit risk decisions, and track corrective actions to verified closure.
Next: Manage change