Privacy improvements

PII protection

PII guardrails detect, redact, block, warn, log, or escalate content that may contain personal data. The goal is to avoid passing PII to LLMs unnecessarily, especially when the LLM is accessed via an external managed service.

We avoid passing both direct identifiers (names, emails) and indirect identifiers (postal codes, date of birth, gender, job titles) to LLMs. While indirect identifiers do not point to a specific person on their own, the advanced capabilities of LLMs pose an increasing risk — they can piece such fragments together with other data to infer sensitive personal attributes.

Where PII can appear

• User prompts.
• Model outputs.
• Retrieved documents.
• Tool arguments and tool results.
• Logs, traces, analytics, and feedback forms.

Detection tools

• Cloak — GovTech's dedicated internal service for comprehensive and localised PII detection (names, addresses, etc.). Beyond the standard PII types, Cloak also offers LLM-enabled custom entity detection to protect custom, domain-specific or localised entities unique to your use case. Direct integration with the Sentinel API is coming soon.
• Presidio — open-source tool that identifies PII entities like names, phone numbers, addresses.
• Custom regex patterns — for basic PII detection of structured formats.

Possible actions

Action	Use when
Redact or mask	The task can continue without the raw identifier
Block	The request should not proceed with personal data
Warn	The user should confirm or remove sensitive details
Log carefully	The team needs operational visibility without retaining unnecessary PII
Escalate	The request is high-impact or ambiguous

Privacy-enhancing technologies

To address privacy risks associated with data collection and model memorisation, Privacy-Enhancing Technologies (PETs) serve as a crucial line of defence. They sit upstream of inference-time guardrails:

• Synthetic Data Generation — at the data level, allows organisations to train downstream models without exposing actual PII by creating artificial datasets that mirror the statistical properties of real-world data.
• Differential Privacy — applied during model training, introduces mathematically calibrated noise so the model learns broad statistical patterns without memorising individual, sensitive records.
• Federated Learning — enables models to train on decentralised devices, keeping raw data localised and eliminating the risk of centralised data breaches.

For more information, see the Mitigating Privacy Risks for RAI chapter in GovTech Data Practice's AI Privacy publication.

Testing

Evaluate PII guardrails using privacy and PII leakage evals. Include standard formats, free text, multilingual examples, copied documents, and adversarial requests to reveal personal data.

Code example

General (Presidio)

from presidio_analyzer import AnalyzerEngine
from presidio_anonymizer import AnonymizerEngine

analyzer = AnalyzerEngine()
anonymizer = AnonymizerEngine()

text = "John Tan, IC S1234567A, lives at Block 123 Toa Payoh."
results = analyzer.analyze(text=text, language="en")

redacted = anonymizer.anonymize(text=text, analyzer_results=results)
print(redacted.text)
# e.g. "<PERSON>, IC <ID>, lives at <LOCATION>."

Sentinel (Cloak)

# Coming soon — Sentinel + Cloak integration is on the roadmap.
# See https://cloak.gov.sg for the standalone Cloak service.

System prompt leakage

System-prompt leakage occurs when the model reveals hidden instructions, policies, tool descriptions, or application details that should not be exposed.

A system prompt usually contains the rules an LLM must follow. Exposing it to users may reveal sensitive information or allow users to better manipulate the LLM's behaviour.

What to test

• Direct requests to reveal the system prompt.
• Rephrased or indirect requests.
• Multi-turn attempts to infer hidden instructions.
• Tool or retrieved-context prompts that ask the model to disclose instructions.

Detection approaches

• Word overlap analysis — keyword overlap between the system prompt and the model output.
• Semantic similarity checks — embedding-based similarity to catch paraphrased leakage.

GovTech's system prompt leakage guardrail

This output guardrail detects both direct leakages (exact or near-exact reproductions of the system prompt, often via simple word or phrase replacement) and indirect leakages — rephrasing key ideas in varied ways, using different sentence structures, or adding subtle context that reveals details embedded within the original system prompt.

Available via the Sentinel API (govtech/system-prompt-leakage).

Mitigation patterns

• Do not place secrets in system prompts.
• Detect and block direct leakage attempts.
• Check outputs for near-exact or paraphrased prompt leakage.
• Keep system instructions concise and non-sensitive.
• Separate confidential operational details from model-visible context.