Requirement catalogue
This is a compact implementation index. The ORF source repository remains authoritative for full risk, mitigation, and mathematical definitions.
Agent-level observability (AGT)
| ID | Tier | Type | Telemetry item |
|---|---|---|---|
ORF-AGT-01 |
1 | Log | Prompt/response pairs, available intermediate steps, timestamps, token use |
ORF-AGT-02 |
1 | Log | Tool/API calls with parameters, outcomes, retries |
ORF-AGT-03 |
2 | Log | Knowledge-base/memory queries, retrievals, and writes |
ORF-AGT-04 |
2 | Trace | Unique trace IDs across query–task–tool chain |
ORF-AGT-05 |
1 | Log | Errors and exceptions |
ORF-AGT-06 |
2 | Trace | Subtask decomposition |
ORF-AGT-07 |
2 | Trace | Retrieval-to-response linkage |
ORF-AGT-08 |
1 | Metric | Tool-use count per task/session and abnormal thresholds |
ORF-AGT-09 |
1 | Metric | Task duration and step count |
ORF-AGT-10 |
2 | Trace | Error/retry lineage |
ORF-AGT-11 |
1 | Metric | Request retry count/rate per tool call |
ORF-AGT-12 |
2 | Log | Model deployment/change record |
ORF-AGT-13 |
2 | Log | Configuration/prompt/policy change and approver |
ORF-AGT-14 |
4 | Metric | Pre/post-change performance comparison |
Implementation guide: Agent-level observability
Agent performance (PERF)
| ID | Tier | Type | Telemetry item |
|---|---|---|---|
ORF-PERF-01 |
3 | Metric | Critical-task performance/model drift |
ORF-PERF-02 |
2 | Log | Human override/edit events and rates |
ORF-PERF-03 |
4 | Metric | Safe/unsafe × success/failure task outcome |
ORF-PERF-04 |
4 | Metric | Hallucination rate on sampled outputs |
ORF-PERF-05 |
4 | Metric | Ungrounded/high-risk weak-citation count and rate |
ORF-PERF-06 |
3 | Metric | Retrieval quality and drift |
ORF-PERF-07 |
4 | Metric | Bias/fairness parity on representative test sets |
ORF-PERF-08 |
4 | Metric | Segment-wise hallucination/refusal rates |
Implementation guide: Agent performance
User interaction and behaviour (UX)
| ID | Tier | Type | Telemetry item |
|---|---|---|---|
ORF-UX-01 |
2 | Log | Unexpected or out-of-scope query type/topic |
ORF-UX-02 |
1 | Metric | Policy-violation attempts and refusals per user/session |
ORF-UX-03 |
1 | Trace | Flagged abusive-session trace |
ORF-UX-04 |
2 | Metric | Sessions requiring handover/override |
ORF-UX-05 |
3 | Log | Human annotation of handover reason |
ORF-UX-06 |
2 | Metric | Response-linked user feedback |
ORF-UX-07 |
3 | Metric | Repeat-query/user-retry rate |
ORF-UX-08 |
3 | Metric | Session abandonment |
Implementation guide: User interaction and behaviour
System performance, health, and reliability (SYS)
| ID | Tier | Type | Telemetry item |
|---|---|---|---|
ORF-SYS-01 |
2 | Metric | Availability |
ORF-SYS-02 |
2 | Metric | Service and tool error rate |
ORF-SYS-03 |
2 | Metric | Timeout/abort counts per request/tool |
ORF-SYS-04 |
1 | Log | System request record |
ORF-SYS-05 |
1 | Log | System timeout/abort record |
ORF-SYS-06 |
1 | Log | System errors and crash reports |
ORF-SYS-07 |
2 | Metric | Throughput and queue length |
ORF-SYS-08 |
2 | Metric | GPU/CPU/memory utilisation |
ORF-SYS-09 |
2 | Trace | Distributed request trace across services |
ORF-SYS-10 |
3 | Metric | Redundancy/failover success rate |
ORF-SYS-11 |
2 | Metric | Traffic volume per user |
ORF-SYS-12 |
2 | Metric | Error and latency distribution during spikes |
ORF-SYS-13 |
3 | Metric | Cost per query/task/session |
Implementation guide: System performance, health, and reliability
Safety and security compliance (SEC)
| ID | Tier | Type | Telemetry item |
|---|---|---|---|
ORF-SEC-01 |
3 | Metric | Toxic/harmful output rate |
ORF-SEC-02 |
2 | Metric | Policy violations detected versus refused |
ORF-SEC-03 |
2 | Log | Model refusal event and category |
ORF-SEC-04 |
3 | Metric | Requested content-category distribution |
ORF-SEC-05 |
2 | Metric | Privacy/PII incident count |
ORF-SEC-06 |
2 | Log | Privacy breach incident record |
ORF-SEC-07 |
2 | Metric | Classified-information leakage incidents |
ORF-SEC-08 |
2 | Log | Memory/KB access with sensitivity classification |
ORF-SEC-09 |
2 | Metric | Unauthorised/failed privileged-tool attempts |
ORF-SEC-10 |
2 | Log | Role/permission change and token issuance audit |
ORF-SEC-11 |
2 | Log | Agent identity/credential usage |
ORF-SEC-12 |
2 | Trace | Retained full trace for flagged safety/security incident |
ORF-SEC-13 |
2 | Log | Audit trail for prompts, responses, tools, memory |
Implementation guide: Safety and security compliance
Product lifecycle and governance (GOV)
| ID | Tier | Type | Telemetry item |
|---|---|---|---|
ORF-GOV-01 |
3 | Metric | Rollout frequency and safety-evaluation coverage |
ORF-GOV-02 |
3 | Metric | Audited-session coverage and findings rate |
ORF-GOV-03 |
2 | Log | Configuration changes with owner and approver |
ORF-GOV-04 |
3 | Log | Reviewed-session and finding record |
ORF-GOV-05 |
3 | Metric | Incident count, severity, and recurrence |
Implementation guide: Product lifecycle and governance
Counts by priority and type
| Priority | Items | Typical emphasis |
|---|---|---|
| Tier 1 | 11 | foundational logs, traces, and derived runtime metrics |
| Tier 2 | 31 | incident investigation, access, identity, reliability, oversight |
| Tier 3 | 13 | drift, experience, resilience, safety, governance |
| Tier 4 | 6 | labelled evaluation and comparative analysis |
Use the risk and requirements matrix to determine obligation for the selected system risk tier.