Skip to content

Investigation — Tier 2

Tier 2 contains 31 items. They are mandatory for High-Risk systems and mandatory but waivable for Medium-Risk systems. This phase extends foundational records into broader investigation and accountability capability.

Agent execution and change

ORF item Implement
ORF-AGT-03 Knowledge/memory query, retrieved document IDs and versions, scores where useful, classifications, and memory writes.
ORF-AGT-04 One trace context across query, task, model, retrieval, tool, and downstream service operations; calculate trace coverage.
ORF-AGT-06 Parent/child task tree with agent, objective, start/end, result, and terminal state.
ORF-AGT-07 Link retrieved evidence to the response or response span it informed.
ORF-AGT-10 Retry lineage linking attempt, failure, next attempt, backoff, and final outcome.
ORF-AGT-12 Model deployment record with actor, model/version, environment, rollout/cohort, and change reference.
ORF-AGT-13 Prompt, configuration, and policy change with artefact version, owner, approver, timestamp, and rationale.

Human oversight and user experience

ORF item Implement
ORF-PERF-02 Human edit/override event linked to the output, with counts and rate. Do not store unnecessary free text.
ORF-UX-01 Out-of-scope or unexpected query classification with approved topic label and rate.
ORF-UX-04 Handover/override request, completion, outcome, queue or destination, and rate per session.
ORF-UX-06 User feedback linked to the exact response/version and protected user/session reference.

System reliability and capacity

ORF item Implement
ORF-SYS-01 Availability over a defined service boundary and observation window.
ORF-SYS-02 Error rate by service, operation/tool, error class, and release.
ORF-SYS-03 Timeout and abort counts per request and tool, including configured limit.
ORF-SYS-07 Throughput and queue depth with saturation thresholds.
ORF-SYS-08 CPU/GPU/memory utilisation where the team operates the capacity.
ORF-SYS-09 Distributed spans with trace/span/parent IDs, service, operation, and duration.
ORF-SYS-11 Traffic volume per protected user reference or approved tenant/account dimension.
ORF-SYS-12 Latency quantiles and error distribution during defined traffic spikes.

Safety, privacy, identity, and audit

ORF item Implement
ORF-SEC-02 Detected policy violations and refused subset; publish refused share with total counts.
ORF-SEC-03 Refusal event and approved category such as safety, privacy, scope, or capability.
ORF-SEC-05 Privacy/PII exposure incident count from governed detection and triage rules.
ORF-SEC-06 Privacy incident record with case, type, severity, trace, containment, and status.
ORF-SEC-07 Classified-information leakage incident count and classification level.
ORF-SEC-08 Memory/KB access event with actor/agent, resource, operation, and sensitivity.
ORF-SEC-09 Unauthorised or failed privileged-tool attempt and denominator of privileged attempts.
ORF-SEC-10 Role/permission change and token issuance audit with actor, target, scope, and authorisation.
ORF-SEC-11 Agent credential usage with non-secret credential ID, scope, tool, and outcome.
ORF-SEC-12 Integrity-protected full approved trace for flagged safety/security incidents.
ORF-SEC-13 Append-only or tamper-evident audit trail for prompts, responses, tools, and memory operations.

Governance change evidence

ORF item Implement
ORF-GOV-03 Configuration change record with owner, approver, artefact, description, timestamp, and release link. Reuse ORF-AGT-13 where the schema meets both definitions.

Investigation views to provide

  1. Session reconstruction: ordered model, retrieval, policy, human, and tool timeline.
  2. Incident view: flagged evidence, severity, affected release, containment, and case links.
  3. Change view: deployment/configuration timeline with before/after incident and reliability signals.
  4. Identity view: agent, credential, permission, and privileged-tool activity.
  5. Reliability view: availability, saturation, errors, timeouts, latency, and affected downstream operations.

Phase exercise

Simulate a retrieved document containing a malicious instruction, followed by a blocked privileged tool attempt, human escalation, and downstream timeout. Confirm the team can determine what was retrieved, which policy acted, which identity attempted the tool, whether external state changed, who reviewed it, and which release was active.

Exit criteria

  • every applicable Tier 2 item is Verified or has a valid approved waiver where allowed;
  • end-to-end trace coverage meets the approved target;
  • privacy, classified-data, credential, and incident evidence follow approved access and retention paths;
  • operators can complete the phase exercise within the investigation-time target; and
  • change, incident, security, and service-management systems contain reciprocal links.

Next: Analytics — Tiers 3 and 4