Further Reading

This list is a work in progress and is not exhaustive.

Use these papers, repositories, and guides to support your evaluation work.

LLM judges

• A Survey on LLM-as-a-Judge reviews methods for building reliable LLM judges.
• Awesome-LLM-Judges collects research on automated evaluation with LLM judges.

Benchmarks and evaluation data

• Holistic Evaluation of Language Models (HELM) provides a framework for evaluating foundation models.
• Awesome Agent Evals collects papers, tools, and benchmarks for agent evaluation.

Practical guides

• Frequently Asked Questions (And Answers) About AI Evals by Hamel Husain answers common questions about evaluation work.
• Beyond vibe checks: A PM's complete guide to evals by Aman Khan in Lenny's Newsletter introduces evaluation for product managers. Full access requires a subscription.
• Evaluating agents by Braintrust covers tests for complete agent workflows and individual steps.
• What are evals? by Arize explains automated evaluation and how it works with human review.