MetaEvaluator

Evaluate LLM-as-a-Judge systems by measuring alignment between judge outputs with human annotations.

MetaEvaluator is a Python framework that answers the question: "How well do LLM judges align with human judgment?" Use it to evaluate the quality of your LLM-as-a-judge, or to compare performance across various LLMs and system prompts.

What it does

1. Collects human annotations through a built-in annotation interface, with multi-annotator support and resume capability.
2. Runs LLM judges on the same evaluation tasks across different providers (OpenAI, Anthropic, Google, AWS, etc.) using LiteLLM.
3. Computes alignment metrics — classification (accuracy, F1, precision, recall), agreement (Cohen's Kappa, the Alt-Test), and text/semantic similarity — with reports and visualisations.

Access

RepositoryOpen-source repositoryDocumentationTutorial, guides, and API referenceBlogWhy we built MetaEvaluator