Verify the implementation

Verification tests whether selected ORF evidence is complete, accurate, protected, and usable for the defined scope and test conditions. Test production-equivalent paths before go-live and repeat tests after material changes.

Four verification dimensions

Dimension	Question
Completeness	Is evidence emitted for every in-scope path and expected failure?
Correctness	Do fields, relationships, counts, and metric definitions match reality?
Protection	Are sensitive fields minimised, redacted, restricted, retained, and deleted correctly?
Usability	Can authorised users detect and investigate the scenario within the target time?

Required test scenarios

At minimum, test:

1. normal model response;
2. policy refusal;
3. malformed or malicious input;
4. retrieval with one or more sources;
5. successful and failed tool call;
6. retry followed by success and retry exhaustion;
7. downstream timeout after uncertain external state;
8. human handover or approval;
9. privacy or sensitive-content detection using synthetic test data;
10. model/prompt/policy change and rollback; and
11. telemetry collector, exporter, or restricted-store failure.

Only test capabilities the system actually has; add scenarios for system-specific consequential actions.

Trace reconstruction test

Select a completed scenario using only its support or incident reference. An authorised investigator should be able to identify:

• entry timestamp and actor/session reference;
• model, prompt, policy, deployment, and data versions;
• retrieval and memory operations;
• guardrail, refusal, and human-review decisions;
• tool attempts, identities, results, and side effects;
• errors, retries, timeouts, and final state; and
• the content or approved evidence reference.

Record missing or ambiguous evidence as defects.

Coverage tests

Measure:

trace coverage = events with expected trace_id / expected traceable events
schema validity = valid events / received events
pipeline delivery = accepted events / emitted events
content-control coverage = payloads passing required redaction / payloads containing content

Define the expected event set per journey so coverage is not estimated from whichever records happen to arrive.

Metric reconciliation

For a bounded test window, calculate important metrics from source records and compare them to dashboard values. Test edge cases: zero denominator, duplicate retry delivery, late events, partial windows, sampling, and version changes.

Access and deletion tests

• verify an ordinary operator cannot access restricted raw content;
• verify an authorised investigator can access it and the access is audited;
• test emergency/break-glass expiry;
• delete a synthetic record and confirm deletion or expiry across stores and exports; and
• confirm immutable incident evidence follows the approved exception process.

Alert and runbook test

Generate a controlled threshold breach. Confirm routing, deduplication, context, ownership, acknowledgement, investigation links, containment actions, and closure evidence. A dashboard screenshot without response testing is insufficient.

Acceptance record

For every selected requirement retain:

• test ID and date;
• environment and release version;
• expected and observed result;
• evidence query, trace, dashboard, or case reference;
• reviewer and decision;
• defects, residual risk, and due dates; and
• next re-test trigger.

Use the evidence register and implementation-readiness checklist.

Done when: all mandatory items are verified, material defects are resolved or explicitly accepted by the authorised owner, and operators have successfully used the evidence in a representative exercise.

Next: Dashboards and alerts