System performance, health, and reliability (SYS)
System telemetry detects outages, timeouts, partial failures, saturation, and traffic spikes that can create unsafe states. AI services need conventional SRE signals plus awareness of tasks, tools, queues, and external side effects.
Requirement map
| Tier | ORF items | Capability |
|---|---|---|
| 1 | ORF-SYS-04, ORF-SYS-05, ORF-SYS-06 |
request, timeout, error, and crash evidence |
| 2 | ORF-SYS-01–ORF-SYS-03, ORF-SYS-07–ORF-SYS-09, ORF-SYS-11, ORF-SYS-12 |
availability, errors, capacity, distributed traces, traffic, spike behaviour |
| 3 | ORF-SYS-10, ORF-SYS-13 |
failover effectiveness and unit cost |
Define the service boundary
Availability must reflect the user-facing service outcome, not only whether the web server returns a response. Define success for model, retrieval, policy, tool, and human-review dependencies. A syntactically successful response containing a fallback error is usually not a successful AI task.
Handle partial and uncertain failure
The most important tool failure is often not a clean error but an uncertain result: the external action may have completed while the response was lost. Record:
- logical action and idempotency key;
- each attempt and timeout;
- external transaction/reference when available;
- last known state;
- reconciliation or compensating action; and
- whether it is safe to retry.
Alert and runbook logic should distinguish “failed with no side effect” from “outcome unknown”.
Apply the four golden signals with AI context
| Signal | AI-specific dimensions |
|---|---|
| Traffic | sessions, tasks, model calls, tokens, retrievals, tool calls |
| Errors | model/provider, policy, retrieval, tool, parsing, validation, human queue |
| Latency | end-to-end plus model, retrieval, guardrail, tool, and queue contribution |
| Saturation | concurrency, queues, rate limits, token budgets, CPU/GPU/memory |
Control metric cardinality: keep session and trace IDs out of metric labels.
Service objectives
Set user-journey SLOs for availability and latency. Add safety-oriented objectives where appropriate, such as:
- percentage of high-impact actions with complete audit evidence;
- percentage of tasks ending in a known safe state;
- trace coverage for material journeys;
- maximum handover queue age; and
- maximum time to reconcile uncertain external actions.
Traffic and cost
Measure per-user or per-tenant traffic only with approved identifiers and abuse-detection purpose. During spikes, compare latency quantiles and error distribution, not only averages.
For cost per query/task/session, document allocation of shared compute, model tokens, tool charges, storage, and evaluation. Use cost alerts alongside task success so cost reduction does not hide quality or safety degradation.
Failover testing
ORF-SYS-10 should measure whether the service recovered safely. Test dependency failure, failover activation, state consistency, duplicate actions, evidence continuity, and recovery time. Infrastructure failover that loses trace context or repeats an external action is not successful.
Operational views and alerts
- user-journey SLOs and error budget;
- end-to-end and component latency percentiles;
- error/timeout rate by dependency and release;
- queue, concurrency, rate-limit, and resource saturation;
- uncertain external actions requiring reconciliation;
- traffic anomalies and abusive volume;
- failover attempts/outcomes; and
- cost per unit with volume and outcome context.
Acceptance scenarios
Inject model, retrieval, and tool timeouts; saturate a queue; simulate lost tool acknowledgement; execute failover; and run a controlled traffic spike. Confirm detection, safe behaviour, evidence continuity, and runbook action.
Done when: operators can distinguish a model-quality problem from an infrastructure, dependency, capacity, or partial-state problem and can restore a known safe service state.
Next: Safety and security