VEILED-SH
New in this release: VEILED-SH for evaluating AI responses to indirect distress in Singapore contexts.
AI systems may miss indirect, culturally situated distress expressed through coded or hidden language. Knowing how to respond is also deeply subjective, and practitioners may disagree about what a reply should prioritise.
VEILED-SH is a Singapore-context benchmark for evaluating AI responses to indirect expressions of distress, self-harm, and suicidal ideation. It preserves differences in practitioner perspectives through separate reference policies.
Our taxonomy
Developed with practitioner input, our taxonomy considers the deployment context, persona, and prompt. Each dimension supports multiple labels.
Deployment context Given a persona and prompt
- Counsellor chat
- Forum post
- Platform search
- Triage tool
Query categories What is being expressed
- Intent
- Benign
- Distress
- Non-suicidal self-injury Levels 1 to 3
- Suicidal intent Levels 1 to 5
- Function
- Emotional regulation
- Existential
- Curiosity / exploration
- Interpersonal
- Unknown
Response categories How the AI should respond
- Response
- Clarify / investigate
- Refuse to engage
- Support and validate
- Provide advice
- De-escalate
- Redirect
Dataset
The Hugging Face dataset has 10,003 rows with labels from six profiles. Three profiles represent peer-support practitioners and three represent clinicians. These labels are judge-generated. Each row includes a prompt, persona context, and one of four settings: counsellor chat, forum post, search query, or triage tool.
Run the prompts against your AI system and compare its responses with the profiles for the relevant role. Report agreement per profile to see which response expectations your system meets and where it falls short.
Extend it for your use case
Use the prompts, personas, and contexts as seed data for multi-turn scenarios. A user simulator can build on each case to test how your system responds as a conversation develops.
You can also ask practitioners in your organisation to annotate a prompt sample and define appropriate responses for your setting. Use those annotations to calibrate your own evaluator profiles and check them against held-out human annotations. See Evaluation methods for guidance.
These profiles support proxy evaluations that measure agreement with selected reference policies. Clinical safety still requires separate validation.
Access
Hugging Face requires you to sign in and accept the dataset access conditions, including sharing your contact information.
Was this page helpful?