Open tools we use and recommend

A framework for large language model evaluations, with datasets, solvers, scorers and agent tooling in one Python package.

A collection of ready-to-run evaluations written for Inspect.

A framework for few-shot evaluation of language models across a large shared set of tasks.

Holistic Evaluation of Language Models: a framework for reproducible, transparent evaluation of language and multimodal models.

An alignment auditing agent that probes a target model through multi-turn conversations with simulated users and tools. Built on Inspect.

Released by Anthropic in October 2025 and transferred to Meridian Labs in May 2026.

Generates an evaluation suite for one behavior you specify, then measures how often and how severely it appears across models.

Released December 2025.

Settings, model organisms and protocols for running AI control experiments.

An LLM vulnerability scanner that probes models for failures such as prompt injection, jailbreaks and data leakage.

The Python Risk Identification Tool, an open framework for security teams to find risks in generative AI systems.

A standardized evaluation framework for automated red teaming and robust refusal.

Mazeika et al., ICML 2024. arXiv:2402.04249

A library for mechanistic interpretability of GPT-style language models: load a model, cache activations, edit them.

Interpret and manipulate the internals of deep learning models, locally or on remote NDIF hardware.

Train and analyze sparse autoencoders on language models.

An open-source interpretability platform for browsing, searching and testing model features.

Train sparse autoencoders and transcoders on transformer activations.

Exercises and course materials from the ARENA alignment research engineering program, covering transformers, interpretability, RL and evals.

The datasets released with "Towards Understanding Sycophancy in Language Models".

Sharma et al., 2023. arXiv:2310.13548

Human preference data from "Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback".

Bai et al., 2022. arXiv:2204.05862