Evaluation Frameworks

The "Pytest for LLMs".

Agent-auditing framework that runs adversarial and operational inspections against AI agents, producing scored JSON and Markdown reports for CLI, CI, and agent-plugin workflows. Apache-2.0 licensed.

Framework for large language model evaluations from the UK AI Security Institute.

Evaluation toolkit for LLMs across multiple backends with reusable tasks, metrics, and result tracking.

Standardized evaluation metrics.

Unified multimodal evaluation toolkit for text, image, video, and audio tasks with 100+ supported benchmarks.

Gym environment for web task automation and agent evaluation. Includes MiniWoB, WebArena, WorkArena, and more. Apache 2.0 licensed.

Evaluation and tracking for LLM experiments and AI agents. Provides feedback functions for measuring quality, relevance, and groundedness with LangChain and LlamaIndex integrations. MIT licensed.

Open-source evaluation library for LLM and agent applications. Built by LangChain with pre-built evaluators for common use cases including RAG, agents, and structured output validation. MIT licensed.

RAG AutoML tool for automatically finding optimal RAG pipelines. Evaluates and optimizes retrieval-augmented generation with AutoML-style automation for your own data and use-case. Apache 2.0 licensed.

Python & JS/TS SDK for running AI-generated code in secure isolated sandboxes. Essential infrastructure for evaluating code-generating LLMs with safe execution environments. Apache 2.0 licensed.

Streamlined and customizable framework for efficient large model (LLM, VLM, AIGC) evaluation and performance benchmarking. One-stop evaluation solution with 80+ benchmarks. Apache 2.0 licensed.

Framework for running agent evaluations and creating/using RL environments. Evaluate arbitrary agents like Claude Code, OpenHands, and Codex CLI. Build and share benchmarks and environments. Apache 2.0 licensed.

Longitudinal evaluation for agent runs: executes task prompts in private fresh-repository attempt workspaces and retains attempt evidence, evaluator provenance, and reports for later re-evaluation. MIT licensed.

Test suite for decision-model servers that implement the Jev API, reporting accuracy and probability calibration and probing prompt injection, option-order sensitivity, irrelevant padding and repeated-call consistency. Apache 2.0 licensed.

Harness that runs the SoloRoleplayer app headless against a local model server and scores roleplay replies for personality lock, head-hopping, repetition and hard-rule adherence. MIT licensed.

Dependency-free checker that flags head-hopping (writing the other character's words, actions or feelings) in roleplay text. MIT licensed.