Benchmark Suites

Contamination-free LLM benchmark with objective ground-truth scoring. ICLR 2025 spotlight paper featuring frequently-updated questions from recent sources. Tests math, coding, reasoning, language, instruction following, and data analysis.

De-facto standard for generative model evaluation.

Holistic Evaluation of Language Models.

Evaluates LLMs on real-world GitHub issues from 15+ Python repositories.

Real-world multi-step agentic benchmark.

Evaluation platform for benchmarking language and multimodal models across large benchmark suites.

Industry-standard ML inference benchmarks with reference implementations for AI accelerators.

Industry-standard ML training benchmarks from MLCommons. Reference implementations for training AI models at scale across image classification, object detection, NLP, and recommendation tasks. Apache 2.0 licensed.

Open-source evaluation toolkit for large multi-modality models (LMMs). Supports 220+ LMMs and 80+ benchmarks including MMMU, MathVista, and ChartQA. Powers the OpenVLM Leaderboard. Apache 2.0 licensed.

Leaderboard comparing LLM performance at producing hallucinations when summarizing short documents. Systematic evaluation of factual consistency across major models. Apache 2.0 licensed.

Continuously updated benchmark with 21,000+ real-world SWE tasks for evaluating agentic LLMs. Decontaminated, mined from GitHub.

Benchmarking system for evaluating LLM models as OpenClaw coding agents. Built with Rust by the kilo.ai team. MIT licensed.

Prompt-adherence leaderboard for text-to-image models. Same prompts run across 60+ frontier models (Flux 2 Max, Nano Banana Pro, Seedream V4, Reve 2.1, Imagen 4 Ultra, Qwen Image 2.0 Pro) and judged by a VLM panel. [Public methodology](https://imagebench.ai/imagebench-v1/methodology), [head-to-head compare pages](https://imagebench.ai/imagebench-v1/compare), and the full [prompt+judgment dataset on Hugging Face](https://huggingface.co/datasets/dh7/imagebench) (CC0).