Manage change
Model, prompt, policy, retrieval, memory, tool, and orchestration changes can alter behaviour without changing the public API. Treat them as controlled production changes.
Classify changes
| Change class | Examples | Minimum response |
|---|---|---|
| Routine | non-behavioural documentation, approved dependency patch | normal engineering controls |
| Behavioural | prompt, model parameter, retrieval ranking, policy threshold | targeted evaluation and monitored rollout |
| Capability | new tool, memory, external action, autonomy, agent delegation | risk/scope reassessment and ORF selection review |
| Data | new knowledge source, classification, user segment, retention | data/privacy assessment and retrieval/evaluation update |
| Emergency | security fix, model/provider outage response | expedited approval with retrospective review and evidence |
Before change
- assign a change ID and owner;
- record affected artefacts and stable before/after versions;
- assess risk, data, and ORF applicability impact;
- select regression, safety, security, and reliability tests;
- define canary/shadow population where appropriate;
- set success, pause, and rollback thresholds; and
- confirm rollback or safe-forward path.
During rollout
Separate the changed cohort in telemetry. Monitor evidence coverage, errors, timeouts, latency, cost, refusals, handovers, critical-task performance, safety signals, and user feedback appropriate to the change. Do not mix versions in an aggregate that prevents comparison.
After rollout
- compare agreed before/after metrics;
- review representative traces and flagged sessions;
- confirm no unexpected data or tool path appeared;
- record final accept/rollback decision;
- update inventories and baselines; and
- schedule longer-window follow-up for slow signals.
Emergency changes
Record what approval was bypassed, why delay would increase risk, who authorised the exception, what monitoring compensates, and when retrospective evaluation/review will occur. Emergency does not mean unlogged.
ORF items supported
Change management relies particularly on ORF-AGT-12, ORF-AGT-13, ORF-AGT-14, ORF-GOV-01, and ORF-GOV-03, with performance, safety, reliability, and user metrics chosen for the affected risk.
Done when: every material production behaviour can be attributed to a known version and approved change, with evidence showing whether rollout thresholds were met.
Next: Measure adoption