User interaction and behaviour (UX)
User-interaction telemetry detects abuse and out-of-scope use, measures refusal and handover, and identifies experience or capability gaps. It must balance useful patterns with privacy and proportionality.
Requirement map
| Tier | ORF items | Capability |
|---|---|---|
| 1 | ORF-UX-02, ORF-UX-03 |
violation/refusal measures and flagged-session reconstruction |
| 2 | ORF-UX-01, ORF-UX-04, ORF-UX-06 |
out-of-scope use, handover, and response-linked feedback |
| 3 | ORF-UX-05, ORF-UX-07, ORF-UX-08 |
handover reasons, repeats, and abandonment |
Use controlled taxonomies
Define bounded labels for:
- intent or task type;
- out-of-scope topic;
- policy-violation attempt;
- refusal reason;
- handover reason;
- feedback reason where collected; and
- session terminal state.
Version taxonomies and keep an other/unknown path. Free-text labels create privacy risk and inconsistent trend data.
Protect user identity
Use the least identifying reference needed for the purpose. Abuse investigation may require a protected account reference; product trend analysis usually requires only session-level or aggregated data. Document who can re-identify pseudonymous records and under what approval.
Do not infer sensitive user attributes from prompts for general analytics. If demographic segment analysis is justified, use a separately governed source and approved linkage.
Design feedback correctly
Tie feedback to a specific response, trace, model/prompt version, and timestamp. Distinguish no feedback from neutral feedback. Where feedback is optional, do not treat response rates as representative of all users.
Define handover
Record:
- trigger and requesting component;
- handover reason;
- destination or queue;
- whether a person accepted it;
- wait time and outcome;
- whether context was successfully transferred; and
- final session state.
A handover request without acceptance can still be an abandoned or unsafe failure.
Define repeat and abandonment
For repeated queries, specify whether matching is exact, semantic, intent-based, or user-initiated retry. Exclude system-generated retries. For abandonment, define the inactivity window and distinguish browser closure, successful silent completion, transfer, error, and unresolved exit where possible.
Operational views
- intents and out-of-scope topics over time;
- violation attempts and refusals per session/account within privacy limits;
- flagged abusive-session investigations;
- handover rate, wait time, outcome, and reason;
- response-linked feedback with coverage;
- repeat-query rate; and
- abandonment by journey step and terminal state.
Alerts
- sharp increase in prompt injection or violation attempts;
- concentrated abusive activity within approved detection scope;
- refusal rate change after model/policy release;
- handover queue failure or sustained high wait time; and
- abrupt increase in repeat or abandonment rate for a critical journey.
Acceptance scenarios
Test an out-of-scope request, a blocked abuse attempt, a valid request incorrectly refused, a completed handover, an unaccepted handover, feedback submission, and a repeated query. Reconcile source events with dashboard rates.
Done when: telemetry explains why users are blocked, repeat, leave, or require help without creating an unjustified surveillance dataset.
Next: System reliability