Tools from GovTech

This list is a work in progress and is not exhaustive.

These tools support evaluation set design and judge validation. The guides linked below are in the Responsible AI Playbook.

Tool	Use it to
Kaleidoscope	Build evaluation sets, define criteria, review outputs, and calibrate automated scoring.
MetaEvaluator	Compare LLM judge outputs with human annotations.
KnowOrNot	Generate cases that test questions outside the system's knowledge base.
METIS & AEGIS	Coming soon! Specifications-driven functionality: evaluate how well an AI system meets requirements. Standards-driven safety and security: evaluate an AI system against centrally defined standards.