Telemetry design worksheet
Use this worksheet after the system boundary and requirements register are approved. Design for the decision or investigation the evidence must support, then define capture, protection, storage, and verification.
Outcome: every selected ORF item has an implementable signal contract and an acceptance test.
1. Design summary
| Field |
Record |
| System/profile version |
Enter: stable system ID and approved profile version |
| System risk tier |
Select: Low / Medium / High |
| Design owner and reviewers |
Enter: engineering owner plus platform, security/privacy, product/risk reviewers |
| Environments |
List: development, test, staging, production, and evaluation environments in scope |
| Telemetry platforms and stores |
List: trace/log/metric platforms and restricted evidence stores |
| Data-handling approval |
Link: approved classification, capture, access, retention, and deletion decision |
| Target implementation phase |
Select: Phase 1 foundation / Phase 2 investigation / Phase 3 analytics |
2. Identifier and trace model
| Identifier |
Creation and propagation |
Format and lifetime |
Sensitive? |
Failure behaviour |
trace_id |
Example: gateway creates; propagates through app, retrieval, model, guardrail, and tool |
W3C trace ID; one material journey |
No |
Alert when coverage falls below target |
session_id |
Enter: creator and participating services |
Enter: random ID and rotation rule |
Assess: Yes / No |
Enter: reject, generate, or metadata-only behaviour |
query_id / task_id |
Enter: creator and parent relationship |
Enter: format and lifetime |
Assess: Yes / No |
Enter: behaviour when missing |
request_id / external action ID |
Enter: component and idempotency relationship |
Enter: format and reconciliation lifetime |
Assess: Yes / No |
Enter: duplicate and unknown-state handling |
user_ref |
Enter: pseudonymisation service and permitted joins |
Enter: rotation and deletion rule |
Usually pseudonymous |
Enter: safe behaviour when unavailable |
3. Requirement signal contract
Use one compact summary row per selected ORF item, then link its detailed schema or metric contract.
| ORF ID |
Signal and source |
Protected content treatment |
Store and retention |
Consumer and action |
Example: ORF-AGT-04 |
Trace spans from gateway, retrieval, model, guardrail, and tool |
IDs and metadata only; no raw prompt in general trace |
Trace store, 30 days |
Incident query; alert below 99.5% journey coverage |
Example: ORF-SEC-03 |
policy.decision.v2 from output guardrail |
Category and evidence reference; restricted content stored separately |
Security log, 180 days |
Safety dashboard; page on control error spike |
Enter: ORF-___ |
Enter: event/metric/trace and producing component |
Describe: allowlist, redaction, tokenisation, or reference |
Enter: approved store and retention |
Enter: dashboard, alert, review, investigation, or audit use |
Detailed contract record
Create one record per ORF ID in the engineering repository.
| Field |
What to record |
| Requirement |
ORF ID, tier/type, matrix action, and register decision |
| Schema or formula |
Versioned event schema, span attributes, or numerator/denominator definition |
| Dimensions and joins |
Allowed dimensions, cardinality limits, trace/session links, and version fields |
| Quality and coverage |
Expected population, completeness target, validation, lateness, and deduplication |
| Protection |
Field allowlist, classification, redaction/tokenisation, encryption, and access role |
| Operational use |
Dashboard, threshold, alert route, saved query, review cadence, and decision owner |
| Acceptance test |
Input/scenario, expected evidence, negative/privacy test, and retained result |
4. Event and schema inventory
| Event/version |
Trigger and required fields |
Optional content/reference |
Validation behaviour |
Owner |
Example: tool.call.v1 |
Start/end; trace, task, tool, attempt, status, latency |
Redacted parameter hash; restricted result reference |
Quarantine invalid event; emit schema-error metric |
Integration platform |
| Enter: event name/version |
List: trigger and mandatory identifiers/version/outcome fields |
List: approved optional fields or references |
Select: Reject / Quarantine / Metadata only |
Enter: producer owner |
5. Metric contract
| Metric |
Formula and eligible population |
Window and dimensions |
Target and coverage |
Owner and action |
| Example: trace coverage |
Events with valid trace ID ÷ expected traceable events |
5 min and 1 hr; service, journey, release |
≥99.5%; denominator coverage reported |
Platform owner; investigate producer or collector loss |
| Example: retry rate |
Retry attempts ÷ logical tool calls |
15 min; tool, error class, release |
Alert >5% with ≥100 calls |
Tool owner; inspect dependency and retry policy |
| Enter: metric |
Define: numerator, denominator, exclusions, and zero-denominator behaviour |
Enter: window, timezone, and permitted dimensions |
Enter: baseline/threshold, sample size, and coverage |
Enter: decision owner and required action |
See Metric examples for complete calculations and interpretation.
6. Content protection design
- Allowlist/denylist: identify permitted fields and content that must never enter general telemetry.
- Redaction/tokenisation: record component, policy version, validation, and failure behaviour.
- Restricted evidence: define store, access workflow, integrity protection, and case-reference pattern.
- Encryption and keys: identify control owner and separation of duties.
- Export and screenshots: prevent restricted content appearing in alerts, tickets, dashboards, or exports.
- Deletion: cover buffers, queues, primary stores, derived metrics, exports, and backups.
- Access review: define access logging, review cadence, and anomalous-access alerting.
7. Pipeline reliability
| Control |
Example target |
Alert and failure behaviour |
Owner |
| Event/trace coverage |
≥99.5% for material journeys |
Alert on breach; mark downstream metrics untrusted |
Platform observability |
| Schema validity |
≥99.99% valid events |
Quarantine invalid events; page on safety-event loss |
Producer plus telemetry platform |
| Collector/export delivery |
No sustained loss; bounded retry queue |
Alert on backlog/drop; use approved buffering |
Platform/SRE |
| Ingestion latency |
p95 under agreed decision latency |
Mark dashboards stale; block consequential automation if required |
Platform/SRE |
| Redaction failures |
Zero uncontained failures |
Stop content export, isolate event, and notify privacy/security |
Security/privacy control owner |
Replace example targets with service-approved values and test the failure behaviour explicitly.
8. Approval
| Reviewer |
Decision |
Date |
Conditions or evidence |
| Technical |
Select: Approve / Revise; enter reviewer |
Enter date |
Link: schema and tests |
| Platform/SRE |
Select: Approve / Revise; enter reviewer |
Enter date |
Link: reliability and capacity evidence |
| Security/privacy |
Select: Approve / Revise; enter reviewer |
Enter date |
Link: data-handling approval |
| Product/risk |
Select: Approve / Revise; enter reviewer |
Enter date |
State: accepted limitations and review trigger |
Next: implement the selected delivery phase and prepare the evidence register.