SWE-bench

Evaluates LLMs on real-world GitHub issues from 15+ Python repositories.

Guide pending

We haven't written this guide yet.

Guides are drafted from the project's own README and metadata. Until one exists, check the product's repository and documentation for authoritative information.

Nearby on the map

Related projects

LiveBench

LiveBench/LiveBench

Contamination-free LLM benchmark with objective ground-truth scoring. ICLR 2025 spotlight paper featuring frequently-updated questions from recent sources. Tests math, coding, reasoning, language, instruction following, and data analysis.

unknown unknownactivity unknown

lm-evaluation-harness (EleutherAI)

EleutherAI/lm-evaluation-harness

De-facto standard for generative model evaluation.

unknown unknownactivity unknown

HELM (Stanford)

stanford-crfm/helm

Holistic Evaluation of Language Models.

unknown unknownactivity unknown

GAIA

Real-world multi-step agentic benchmark.

unknown unknownactivity unknown

OpenCompass

open-compass/opencompass

Evaluation platform for benchmarking language and multimodal models across large benchmark suites.

unknown unknownactivity unknown

MLPerf Inference

mlcommons/inference

Industry-standard ML inference benchmarks with reference implementations for AI accelerators.

unknown unknownactivity unknown