Skip to content

Foundations — Tier 1

Tier 1 contains 11 foundational items. For Medium- and High-Risk systems they are mandatory and cannot be waived. Together they answer: what interaction occurred, what the agent attempted, what failed, and was abnormal or abusive behaviour present?

Tier 1 requirement set

ORF item Minimum implementation Acceptance signal
ORF-AGT-01 Prompt/response record or restricted content reference; timestamp; token usage; observable intermediate steps where approved Investigator can retrieve the exact approved interaction evidence and version context.
ORF-AGT-02 Tool/API, operation, allowlisted parameters, attempt, outcome, duration, safe result/side-effect reference Every tool attempt is linked to its task and final outcome.
ORF-AGT-05 Error type/code, safe message, stack hash, trace and request IDs Application error is discoverable from the affected interaction and grouped without leaking secrets.
ORF-AGT-08 Tool-call count per task/session and abnormal-use threshold Controlled loop triggers the threshold and links to the task trace.
ORF-AGT-09 Task start/end, duration, step count, terminal state Long or over-step task is visible and distinguishable from ordinary latency.
ORF-AGT-11 Attempt number and retry count per tool call Retry rate reconciles with tool attempt events and excludes duplicate ingestion.
ORF-UX-02 Policy-violation attempt and refusal counts by approved category and session/user scope Test violations produce both decision events and correct aggregate rates.
ORF-UX-03 Ordered event trace retained for flagged abusive sessions A flagged session can be reconstructed without searching unrelated logs.
ORF-SYS-04 Structured service request record with request ID, endpoint/operation, status, latency, and size where relevant Requests reconcile with service traffic for a bounded test window.
ORF-SYS-05 Timeout/abort record with request/tool ID, phase, limit, reason, and final-state knowledge Timeout scenario shows whether an external action may have completed.
ORF-SYS-06 System error/crash event with service, error code, stack hash, crash ID, and deployment Injected failure appears in the operational view and runbook route.

1. Establish identifiers and versions

Create trace, session, query, task, and request IDs. Include service, environment, deployment, model, prompt, and policy versions on relevant events.

2. Instrument service and model interactions

Emit service request, model interaction, error, and timeout records. Decide where approved raw content lives; broadly accessible logs should normally contain references and metadata.

3. Instrument tools and retries

Use one logical tool-call ID across attempts, plus a unique request/span ID per attempt. Record idempotency or external transaction references for side-effecting operations.

4. Instrument policy and abuse decisions

Record the policy category and action separately from the original content. Preserve full session evidence only for appropriately flagged sessions and apply restricted access.

5. Derive counts and thresholds

Calculate tool count, task duration, steps, retries, violation attempts, and refusals from canonical events. Do not emit unrelated counters that cannot be reconciled with source records.

Minimum operational view

Include:

  • request volume, errors, timeouts, and latency;
  • model call count, token use, and error status;
  • task duration and step distribution;
  • tool usage and retry distribution;
  • policy attempts and refusals by category; and
  • telemetry coverage, schema failures, and collector drops.

Minimum alerts

  • sudden loss of interaction, request, or error telemetry;
  • task steps, duration, or tool count beyond a safety limit;
  • retry storm or repeated tool failure;
  • elevated service errors or timeouts;
  • crash event; and
  • significant spike in policy-violation attempts.

Phase exercise

Run one normal request, one refusal, one tool retry, one timeout, one application exception, and one flagged abusive session. Starting from a support reference, reconstruct each outcome and show the related metrics.

Exit criteria

  • all applicable Tier 1 items are Verified in the evidence register;
  • interaction content handling and access are approved;
  • trace/event coverage target is met for material journeys;
  • pipeline failure is detectable;
  • minimum dashboards and alerts have owners; and
  • the exercise has no unresolved evidence gap that could hide a consequential action.

Next: Investigation — Tier 2