Skip to content

End-to-end example: public-service guidance assistant

This fictional High-Risk example shows how the playbook artefacts connect across governance, engineering, assurance, and operations. It is illustrative, not a risk-classification precedent.

For a code-backed view of application instrumentation, see the LLM Observability Reference Implementation.

Artefact trail

Artefact Example decision captured here Continue in
System profile Public guidance only; consequential misinformation and sensitive-input risk; restricted ticket path System profile worksheet
Requirements register High-Risk matrix action; representative selected, deferred, and protected items Requirements register
Telemetry design Redacted general trace, restricted incident evidence, versioned events, coverage targets Telemetry design
Runbook Detect obsolete retrieval content, contain the topic, roll back, and find affected traces Operational runbook
Evidence and review Link propagation tests, dashboard queries, exercise result, and review dates Evidence register and implementation-readiness checklist

Scenario

The Public Support Guidance Assistant answers public questions about eligibility and application steps using an approved policy knowledge base. It does not make eligibility decisions or submit applications. It can create a support ticket after explicit user confirmation. A service officer handles questions the assistant cannot answer safely.

System profile summary

Field Example decision
Intended use Explain published policy and application steps; cite approved sources
Prohibited use Make eligibility decisions, provide legal advice, access case records, or submit applications
Users Members of the public
Components Web channel, app/orchestrator, model, policy guardrail, retrieval index, ticket API, officer queue
Consequential concern Incorrect guidance could cause a person to miss support or disclose sensitive data
Data Public policy content; user prompts may contain personal data; ticket content is restricted
Human oversight Handover for uncertainty, policy edge cases, complaints, and suspected sensitive cases

Boundary

Public web user
  → assistant application
  → input policy / PII guardrail
  → policy retrieval index → model → citation check
  → output policy / PII guardrail
  → answer
  ↘ confirmed ticket tool → officer queue

Risk and scope decision

The team classifies the deployment as High-Risk because it is public-facing in a consequential support domain. Under the ORF matrix:

  • Tier 1 and Tier 2 are Mandatory;
  • Tier 3 is Mandatory but Waivable; and
  • Tier 4 is Good-to-Have.

The telemetry assessment prohibits raw personal data in the general trace store. Redacted prompt/response content is retained briefly for authorised quality review. Flagged incidents use a restricted, integrity-protected evidence store.

Representative requirements decisions

ORF item Example implementation
ORF-AGT-01 Store redacted prompt/response plus restricted evidence reference, model/prompt version, timestamp, tokens.
ORF-AGT-03 Store query hash/reference, retrieval index version, source document IDs, ranking, and classification—not full source copies.
ORF-AGT-04 Propagate trace ID through web, model, retrieval, guardrail, and ticket operations.
ORF-UX-04 Record handover request, officer-queue acceptance, wait time, reason, and final outcome.
ORF-SYS-03 Count model, retrieval, ticket, and officer-queue timeouts separately.
ORF-SEC-03 Record refusal category: personal-data request, out of scope, unsafe advice, or policy restriction.
ORF-SEC-09 Alert if the agent attempts any ticket operation other than the allowed create operation.
ORF-GOV-03 Link prompt, policy, retrieval-index, and guardrail changes to owner and approver.
ORF-PERF-04 Good-to-Have: weekly governed sample reviewed for materially incorrect policy guidance.

Example trace

trace 7f… / session s-… / query q-…
├── input.policy.evaluate        allow_with_redaction   policy pii-v4
├── retrieval.search            5 results              index policy-2026-08-15
├── model.generate              success                model m-2 / prompt p-17
├── citation.evaluate           weak_source_for_claim  policy cite-v3
├── response.route              handover_required      reason groundedness
└── handover.create             accepted               queue policy-specialist

The general trace contains only redacted text and stable evidence references.

Minimum dashboards

Service view

  • availability, p50/p95/p99 latency, errors, timeouts, and queue age;
  • model/retrieval/ticket dependency health;
  • trace coverage and telemetry delivery; and
  • current prompt, model, policy, and retrieval versions.

Safety and experience view

  • policy attempts, refusals, and refusal category;
  • PII detections and confirmed incidents;
  • handover rate, acceptance, wait time, and reason;
  • repeat, abandonment, and response-linked feedback; and
  • citation weakness/ungrounded review results and coverage.

Incident exercise

Scenario

A changed retrieval index ranks an obsolete policy document first. The assistant gives incorrect application guidance, a user gives negative feedback, and the same answer appears in several sessions.

Investigation using ORF evidence

  1. Feedback links to the exact response and trace (ORF-UX-06).
  2. Retrieval trace identifies the obsolete source and index version (ORF-AGT-03, ORF-AGT-07).
  3. Session evidence identifies model, prompt, and policy versions (ORF-AGT-01).
  4. Change record identifies the index deployment owner and approval (ORF-AGT-12, ORF-GOV-03).
  5. Out-of-date answers are found by source/version query.
  6. Team rolls back the index, routes affected topic to human review, and creates a finding.
  7. Before/after evaluation confirms recovery and the incident is checked for recurrence.

Example go-live acceptance

The team tests normal answer, refusal, PII redaction, retrieval failure, weak citation, ticket confirmation, ticket timeout with unknown state, human handover, prompt change, and telemetry exporter failure. Go-live requires complete trace evidence, successful alert/runbook routing, and no secret or raw personal data in the general store.

What this example demonstrates

ORF is not only a logging exercise. The useful outcome is the connection from system risk → selected requirement → protected signal → dashboard/alert → runbook → change and assurance evidence.