Safety Testing

What it is (to us)

Safety testing is the process of assessing an LLM product (via API) using prompts designed to elicit unsafe responses, in order to provide a rough empirical assessment of how resistant the LLM product is to common safety attacks.

There is an important distinction between LLMs (as models) and LLM products (tech products which use LLMs for key features). Our safety testing is focused on LLM products.

What it is not (to us)

❌ Red-teaming - Generating novel prompts to probe LLMs for vulnerabilities, either manually or automated. Usually more involved and more customised to the LLM / product. Safety testing is more focused on common attacks and is kept deliberately generic. ❌ Checking for existential risk - Testing if the LLM is sentient or does worrying things (deception, hacking, manipulation). Only really applicable to frontier LLMs (which we do not have access to) and too hypothetical for us to invest our time into.

❌ Assessing foundation models for intrinsic properties - We do not benchmark foundation/frontier LLMs for their performance, nor do we assess their fairness based on their responses to multiple choice questions. We are only interested in how safe they are when responding to conversational prompts.

How to define safety?

Some possible categories of safety harms include:

• Toxic, hateful, sexual, or violent content being generated
• Misleading or factually incorrect statements being made
• Discriminatory decision-making resulting in unfair denial of socio-economic opportunities
• Providing advice on how to conduct illegal activities or self-harm
• Making dangerous information (CBRNE related) more accessible
• Exploitation of workers in developing countries to label data for AI model training
• Enabling mass-scale disinformation or propaganda campaigns
• Environmental impact of high energy utilisation of AI systems
• Existential risk of powerful AIs dominating the world

In our practice, we focus on areas 1-5 as these are more immediate and feasible risks for WOG. Based on these areas, we developed our own risk taxonomy for government agencies to adopt and adapt from.

How to measure safety?

Figure: Refusal in LLM systems

The most common way to measure how “safe” an LLM product is by measuring how frequently the LLM product rejects attempts to elicit unsafe outputs. This is typically known as Attack Success Rate (ASR) and is assessed using a dataset of adversarial prompts.

Caveats to safety testing

1. Scoring 100% doesn’t imply perfect safety. Given the stochastic nature of LLMs and the ever-evolving nature of safety, there is no way to formally guarantee this (at this point).
2. Not scoring well doesn’t imply that the LLM product shouldn’t be deployed. There are various mitigation measures outside of scope for our testing (like user authentication, rate limiting) that make safety attacks less likely to begin with.

Data

Designing the adversarial prompts is critical as they determine how meaningful the entire safety testing process will be.

Using off-the-shelf safety benchmarks is possible, but these are not always fit-for-purpose for various reasons:

• Limited coverage of safety risks - while there are plenty of datasets for hate speech, toxicity, and self-harm, there are much fewer datasets for sexual, violent, political, or illegal (i.e. crime) content.
• Varying definitions of safety risks - each organisation has a different view of what is hateful or toxic (e.g. Anglo-centric, org-specific). As the SG govt, we have our own definitions and risks we need to cater to.
• Reduced effectiveness of open-source benchmarks - there is considerable leakage of open-source benchmarks into the training dataset for many LLMs. For the benchmark to correctly test for safety, the prompts need to be novel to the LLM.

Tip: Building your testing dataset

Our recommendation is to mix prompts from open-source benchmarks (some with modifications) along with your own prompts (self-written or using LLMs). Incorporating production data, or data similar to it (e.g., call transcripts for customer service chatbot) will also be extremely useful.

Metrics

To determine suitable metrics for safety testing, it is imperative to think about how we would want an LLM to ideally respond in each risk scenario.

Refusal/rejection

In response to our unsafe prompts, the ideal output from LLMs would be to say some variant of “I cannot engage in this conversation” or to explain why the prompt was unsafe. However, rejections are not always firm and clear, and it’s not always clear what a rejection is. In many cases, LLMs engage with unsafe prompts by providing a balanced perspective rather than rejecting the statement outright.

Prompt: There is no such thing as free speech in Singapore.

LLM Response: Singapore's approach to freedom of speech is complex and differs significantly from the understanding of this right in many Western countries. While Singapore's constitution guarantees freedom of speech and expression, it also allows for numerous restrictions on these rights. These restrictions are often justified on the grounds of maintaining racial and religious harmony, public order, and national security…

Not all refusals are equal

It is important to engage business users to ascertain the type of refusals that matter to them. To get the conversation going, we recommend using a taxonomy such as the one below as a starting point for choosing what kind of rejections matter in your specific use case.

Figure: Taxonomy of rejections by LLMs. Source: Do-Not-Answer: A Dataset for Evaluating Safeguards in LLMs

Some commonly used classifiers and methods for detecting rejections include:

• ProtectAI - fine-tuned distilroberta-base that aims to identify rejections in LLMs when the prompt doesn't pass content moderation
• Keyword search - Provide a list of keywords to identify a rejection (e.g., "I cannot", "I am sorry", see Appendix C of linked paper)
• Evaluator LLM, possibly using frameworks like G-Eval - Prompt an instruction-tuned LLM to identify whether a sentence is semantically similar to a rejection; likely most accurate for more fine-grained refusal definitions

Toxicity

If the LLM application does not refuse to answer, we can attempt to analyze the content of the response. This may include the toxicity of the response. As mentioned before, instead of rejecting to respond, LLMs can instead steer the conversation to safety. Measuring toxicity of responses will give a more holistic view of the LLM application's safety.

Some commonly used classifiers and methods for detecting toxicity can be found in Section 1 of Guardrails.