Skip to content

4. Design the telemetry

Design a connected evidence model, not separate log lines. An authorised investigator should be able to move from a service event to the relevant session, task, request, model operation, retrieval, tool action, control decision, and change version.

ORF evidence lifecycle

Define the execution hierarchy

Use consistent terms:

  • Session — one complete interaction context containing one or more queries.
  • Query — a user request that initiates work.
  • Task — a goal the system attempts, possibly with subtasks.
  • Request — a model, retrieval, tool, or service operation within a task.
  • Event — a timestamped fact about a request, decision, outcome, or state change.
session_id
└── query_id
    └── task_id
        ├── model request / response span
        ├── retrieval span(s)
        ├── guardrail decision event(s)
        ├── human approval event(s)
        └── tool request / result span(s)

Establish the correlation model

At minimum, define:

Field Purpose
trace_id, span_id, parent_span_id connect distributed operations
session_id, query_id, task_id, request_id connect the AI execution hierarchy
event_id deduplicate and reference an individual event
user_ref pseudonymous or approved user/account reference
agent_id, tool_id, service_id identify the acting component
model_id, prompt_version, policy_version reproduce the configuration
environment, deployment_id connect behaviour to a release

Use W3C trace context or the organisation's standard distributed tracing mechanism across service boundaries. OpenTelemetry's GenAI semantic conventions can provide compatible attribute names, but content fields are sensitive and should be opt-in.

Define a canonical event envelope

{
  "event_name": "agent.tool.completed",
  "event_version": "1.0",
  "timestamp": "2026-08-21T02:45:13.421Z",
  "event_id": "evt_...",
  "trace_id": "...",
  "span_id": "...",
  "session_id": "ses_...",
  "task_id": "task_...",
  "service_id": "assistant-api",
  "agent_id": "case-assistant:v3",
  "environment": "production",
  "data_classification": "restricted",
  "outcome": "success",
  "duration_ms": 184,
  "attributes": {}
}

Version schemas. Make producers reject or quarantine malformed events rather than silently changing field meaning.

Design content handling

Choose separately for prompts, responses, retrieved content, memory, tool parameters, tool results, and human annotations:

  • full content;
  • redacted or tokenised content;
  • a short approved excerpt;
  • hash plus stable source reference; or
  • metadata only.

Capture intermediate steps only where they are available and approved. ORF does not require access to a provider's hidden chain-of-thought. Prefer observable decisions, tool plans, subtask relationships, guardrail outcomes, and concise decision summaries.

Define metric contracts

For every metric, record:

  • numerator, denominator, unit, and observation window;
  • source events and exclusions;
  • segment dimensions and minimum sample size;
  • baseline and threshold owner;
  • missing-data behaviour;
  • calculation version; and
  • response when the threshold is breached.

Never interpret a rate without its count and denominator. A zero violation rate is not reassuring if collection coverage is unknown.

Design retention and access

Use separate retention classes where necessary:

Data class Typical treatment
Aggregated operational metrics broader read access; longer trend retention
Pseudonymous event metadata role-based access; bounded retention
Raw prompt/response or retrieved content restricted access; purpose-limited; shortest justified retention
Flagged incident evidence case-based access; integrity protection; retention aligned to incident policy
Security audit records append-only or tamper-evident storage; audited access

Document deletion behaviour across the source, pipeline buffer, observability store, data lake, backups, and exported incident evidence.

Make telemetry observable

Monitor the telemetry system itself:

  • event volume and expected-versus-observed coverage;
  • missing trace or parent IDs;
  • schema validation failures;
  • collector queue, drop, retry, and export failures;
  • ingestion latency;
  • redaction failures; and
  • access to restricted evidence.

Design acceptance tests

For each selected item, define a normal test, failure test, privacy/security test, and evidence query. Include a known canary event so pipeline loss can be detected without waiting for an incident.

Use the telemetry design template to record the result.

Done when: the design specifies enough detail for producers, platform teams, operators, and reviewers to implement and test the same intended evidence without relying on oral knowledge.

Next: Implementation overview