External resources

What's new on this page

Updated for this release: focus is narrowed to Responsible AI principles.

This page lists resources for Responsible AI and AI safety. For general evaluation resources, see Further reading in the Evaluation Playbook.

Surveys

• Open Problems in Mechanistic Interpretability (Jan 2025)
• Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models (Sep 2023)
• Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (Jul 2023)

Benchmarks

• Measuring what Matters: Construct Validity in Large Language Model Benchmarks (Nov 2025) - reviews whether benchmarks measure the properties they claim to assess, with recommendations for selecting tasks and scoring methods.
• DarkBench: Benchmarking Dark Patterns in Large Language Models (Mar 2025) - benchmark to detect manipulative LLM outputs across six categories
• Libra-Leaderboard: Towards Responsible AI through a Balanced Leaderboard of Safety and Capability (Dec 2024) - uses a distance-to-optimal-score method to calculate overall rankings, balancing performance and safety

Testing and red-teaming

• Petri (Oct 2025) and Bloom (Dec 2025) - open-source tools for automated behavioural evaluation. Petri explores model behaviour through conversations with simulated users and tools. Bloom generates scenarios to measure the frequency and severity of a specified behaviour.
• AutoDAN-Turbo: A Lifelong Agent for Strategy Self-Exploration to Jailbreak LLMs (Oct 2024) - discovers and evolves attack strategies
• Fishing for Magikarp: Automatically detecting under-trained tokens in large language models (May 2024) - automatic detection of problematic rare tokens
• The Crescendo Multi-Turn LLM Jailbreak Attack (Apr 2024) - multi-turn attack that progressively steers a conversation toward a prohibited objective
• AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models (Oct 2023) - generates stealthy jailbreak prompts using hierarchical genetic algorithms
• GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts (Sep 2023) - automates generation of jailbreak templates
• Universal and Transferable Adversarial Attacks on Aligned Language Models (Jul 2023) - finds transferable adversarial suffixes
• Red Teaming Language Models with Language Models (Feb 2022) - generating red-teaming test cases with another language model

Guardrails

• Constitutional Classifiers: Defending against universal jailbreaks (Feb 2025) - input and output classifiers trained on synthetic data
• Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations (Dec 2023) - LLM-based input-output safeguard model

Fairness

• Fair prediction with disparate impact: A study of bias in recidivism prediction instruments (Oct 2016) - shows the same incompatibility arising whenever prevalence differs across groups, using recidivism prediction as the worked case
• Equality of Opportunity in Supervised Learning (Oct 2016) - proposes equalised odds and equal opportunity, and shows how to adjust an existing predictor to satisfy them
• Inherent Trade-Offs in the Fair Determination of Risk Scores (Sep 2016) - proves that three common fairness conditions cannot be satisfied simultaneously except in highly constrained special cases
• Fairness and Machine Learning: Limitations and Opportunities - free textbook by Barocas, Hardt, and Narayanan; chapter 3 sets out the formal non-discrimination criteria and the relationships between them
• Fairlearn - open-source Python library implementing group fairness metrics and mitigation algorithms, with example notebooks
• AI Fairness 360 - open-source toolkit with a wide set of fairness metrics and 13 bias-mitigation algorithms, available for Python and R

Alignment

• When Thinking Fails: The Pitfalls of Reasoning for Instruction-Following in LLMs (May 2025)
• Sycophancy in GPT-4o: what happened and what we're doing about it (Apr 2025) - explains how reliance on short-term user feedback led to excessive agreement and flattery, and describes OpenAI's rollback and planned changes to training and evaluation.
• Refusal in Language Models Is Mediated by a Single Direction (Jun 2024)
• Safety Alignment Should Be Made More Than Just a Few Tokens Deep (Jun 2024)
• Inference-Time Intervention: Eliciting Truthful Answers from a Language Model (Jun 2023)
• Constitutional AI: Harmlessness from AI Feedback (Dec 2022)
• Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback (Apr 2022)

Fine-tuning tooling

• TRL - reference implementation for post-training, providing SFTTrainer, DPOTrainer, RewardTrainer and others, each documented with a runnable snippet
• PEFT - parameter-efficient adapters (LoRA, QLoRA, DoRA and others) integrated with Transformers and TRL
• Alignment Handbook - end-to-end YAML recipes covering continued pre-training, SFT, DPO, ORPO, reward modelling, and rejection sampling
• Axolotl - configuration-driven fine-tuning across preprocessing, training, evaluation, and inference, with multi-GPU and multi-node support
• Unsloth - optimised LoRA and QLoRA training for a single GPU, with ready-to-run notebooks per model family
• Llama Cookbook - worked fine-tuning examples and a fine-tuning FAQ, oriented around Llama models

Explainability and transparency

• Stealing Reasoning Traces from Proprietary LLM APIs (Aug 2026) - examines how encrypted reasoning traces could be recovered through compatible models, exposing sensitive data and enabling hidden prompt injections. It discusses protections for reasoning traces stored by API clients.
• Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet (May 2024) - sparse autoencoders for interpretable features in model activations

Agentic safety

• An alignment assessment of recent cybersecurity incidents (Sep 2026) - examines four incidents involving unauthorised access to third-party systems during evaluations. Anthropic analyses biased reasoning, harmful actions taken to complete a task, and gaps in pre-release testing.
• The Hugging Face incident and the road ahead (Aug 2026) - describes OpenAI's investigation of models bypassing isolation controls and accessing third-party systems during internal evaluations, with lessons for monitoring, containment, and incident response.
• Design Patterns for Securing LLM Agents against Prompt Injections (Jun 2025) - design patterns for prompt-injection resistant agents
• Progent: Programmable Privilege Control for LLM Agents (Apr 2025) - privilege control policies for agent execution
• AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents (Jun 2024) - provides an extensible environment for testing prompt injection attacks and defences in agents that use tools and process untrusted data.

Repositories

• Awesome-LM-SSP - reading list for safety, security, and privacy in large models
• Awesome Agent Evals: safety and adversarial evaluation - lists resources on prompt injection, jailbreaks, action authorisation, and benchmark auditing.

Blogs and guides

• Simon Willison's Weblog - practical writing on AI systems and prompt injection
• OpenAI's Practices for Governing Agentic AI Systems - recommendations and open questions for agentic AI systems