Evaluation, Benchmarks & Datasets

Curated 15T+ token web dataset for pre-training.

Multimodal agent benchmark dataset.

Contamination-free LLM benchmark with objective ground-truth scoring. ICLR 2025 spotlight paper featuring frequently-updated questions from recent sources. Tests math, coding, reasoning, language, instruction following, and data analysis.

De-facto standard for generative model evaluation.

Holistic Evaluation of Language Models.

Evaluates LLMs on real-world GitHub issues from 15+ Python repositories.

Real-world multi-step agentic benchmark.

Evaluation platform for benchmarking language and multimodal models across large benchmark suites.

Industry-standard ML inference benchmarks with reference implementations for AI accelerators.

Industry-standard ML training benchmarks from MLCommons. Reference implementations for training AI models at scale across image classification, object detection, NLP, and recommendation tasks. Apache 2.0 licensed.

Open-source evaluation toolkit for large multi-modality models (LMMs). Supports 220+ LMMs and 80+ benchmarks including MMMU, MathVista, and ChartQA. Powers the OpenVLM Leaderboard. Apache 2.0 licensed.

Leaderboard comparing LLM performance at producing hallucinations when summarizing short documents. Systematic evaluation of factual consistency across major models. Apache 2.0 licensed.

Continuously updated benchmark with 21,000+ real-world SWE tasks for evaluating agentic LLMs. Decontaminated, mined from GitHub.

Benchmarking system for evaluating LLM models as OpenClaw coding agents. Built with Rust by the kilo.ai team. MIT licensed.

Prompt-adherence leaderboard for text-to-image models. Same prompts run across 60+ frontier models (Flux 2 Max, Nano Banana Pro, Seedream V4, Reve 2.1, Imagen 4 Ultra, Qwen Image 2.0 Pro) and judged by a VLM panel. [Public methodology](https://imagebench.ai/imagebench-v1/methodology), [head-to-head compare pages](https://imagebench.ai/imagebench-v1/compare), and the full [prompt+judgment dataset on Hugging Face](https://huggingface.co/datasets/dh7/imagebench) (CC0).

The "Pytest for LLMs".

Agent-auditing framework that runs adversarial and operational inspections against AI agents, producing scored JSON and Markdown reports for CLI, CI, and agent-plugin workflows. Apache-2.0 licensed.

Framework for large language model evaluations from the UK AI Security Institute.

Evaluation toolkit for LLMs across multiple backends with reusable tasks, metrics, and result tracking.

Standardized evaluation metrics.

Unified multimodal evaluation toolkit for text, image, video, and audio tasks with 100+ supported benchmarks.

Gym environment for web task automation and agent evaluation. Includes MiniWoB, WebArena, WorkArena, and more. Apache 2.0 licensed.

Evaluation and tracking for LLM experiments and AI agents. Provides feedback functions for measuring quality, relevance, and groundedness with LangChain and LlamaIndex integrations. MIT licensed.

Open-source evaluation library for LLM and agent applications. Built by LangChain with pre-built evaluators for common use cases including RAG, agents, and structured output validation. MIT licensed.