Agent-level observability (AGT)

Agent-level observability makes an individual query, task, decision path, retrieval, and tool action reconstructable. It addresses incomplete evidence, runaway behaviour, and silent regressions after model, tool, prompt, or policy changes.

Fictional trace view showing connected execution, evidence references, versions, and safe terminal state. Select the image to open it at full size.

Requirement map

Tier	ORF items	Capability
1	ORF-AGT-01, ORF-AGT-02, ORF-AGT-05, ORF-AGT-08, ORF-AGT-09, ORF-AGT-11	interaction/tool/error records and loop indicators
2	ORF-AGT-03, ORF-AGT-04, ORF-AGT-06, ORF-AGT-07, ORF-AGT-10, ORF-AGT-12, ORF-AGT-13	retrieval, distributed trace, task tree, retry lineage, and change evidence
4	ORF-AGT-14	pre/post-change performance comparison

Instrument the agent execution model

Represent an agent task as connected spans or events rather than one opaque model call.

Operation	Important fields
Task	task/parent IDs, objective category, agent version, start/end, terminal state
Model	model ID, prompt/policy version, operation, tokens, duration, result status
Retrieval	query reference, collection/index version, document IDs, scores, classification
Tool attempt	tool/operation, safe parameters, credential scope, attempt, result, side-effect reference
Decision/control	policy version, category, decision, resulting route
Change	artefact, old/new version, actor, approver, release, timestamp

Control runaway behaviour

Record configured limits alongside observations: maximum steps, task duration, tool calls, retries, concurrent work, and cost where used. An alert without an enforcement control only reports the runaway task; pair metrics with circuit breakers and an interrupt or safe-handover path.

Distinguish:

• a retry of the same logical call;
• a new call with changed parameters;
• a compensating action; and
• a duplicated event caused by telemetry delivery.

For side-effecting tools, capture the idempotency or transaction reference and whether final external state is known.

Handle intermediate reasoning carefully

ORF-AGT-01 refers to intermediate reasoning steps where available. Capture observable subtask structure, tool plans, policy decisions, and short system-generated decision summaries where approved. Do not attempt to extract hidden provider chain-of-thought, and do not store sensitive reasoning content merely because an SDK exposes it.

Operational views

• session/task timeline;
• task duration and step distributions by agent/release;
• tool calls and retries by tool/outcome;
• errors grouped by stack hash and deployment;
• trace coverage and orphan-span rate;
• retrieval sources used per response; and
• change timeline overlaid with errors, refusals, and incidents.

Alerts

• step, duration, retry, or tool-call safety limit reached;
• repeated identical failure or retry lineage;
• missing trace context above threshold;
• abrupt loss of prompt/response or tool telemetry;
• unapproved model/prompt/policy version in production; and
• significant post-change regression against a predefined guardrail.

Acceptance scenarios

1. A task calls a tool twice after a transient failure; show both attempts and one logical call lineage.
2. A task exceeds its step limit; show the threshold, circuit-breaker decision, and safe terminal state.
3. A retrieval-grounded response is challenged; show the source IDs and response linkage.
4. A prompt version changes; show actor, approver, release, and relevant before/after signals.

Evidence of good implementation

An investigator can reconstruct the ordered agent path without relying on raw application console output, and the team can distinguish what the agent intended, attempted, observed, and ultimately changed.

Relevant ARC themes include comprehensive logging, distributed tracing, immutable audit logs, circuit breakers, reasoning limits, interruptability, and periodic audits. See the ORF source crosswalk for authoritative mapping and mapping basis.

Next: Agent performance