Results for "evals"

12 in-depth AI engineering project briefs with staged milestones, evals, and deploy steps. MIT licensed.

Comprehensive set of tools to assess and improve LLM security. Includes Llama Guard safety classifiers, CyberSec Eval benchmarks, and Prompt Guard for prompt injection detection. BSD-3-Clause licensed.

Unified multimodal evaluation toolkit for text, image, video, and audio tasks with 100+ supported benchmarks.

85 buildable AI engineering project briefs grouped by theme (RAG, agents, evals, fine-tuning, multimodal), each with difficulty, skills shown, and reference resources. MIT licensed.

Streamlined and customizable framework for efficient large model (LLM, VLM, AIGC) evaluation and performance benchmarking. One-stop evaluation solution with 80+ benchmarks. Apache 2.0 licensed.

Democratizing Reinforcement Learning for LLMs. Framework for training AI agents with RL featuring near-zero code changes, CLI-first workflow, and 50+ built-in benchmarks. Supports GRPO, REINFORCE, RLOO with verl and tinker backends. Apache 2.0 licensed.

Production SDK and harness runtime for building, monitoring, and controlling end-to-end AI agent lifecycles in Python and TypeScript. Apache 2.0 licensed.

One-config CLI for LLM post-training: SFT, DPO, GRPO, KTO, ORPO, plus eval gating and export. Layer streaming trains an 8B model on a 4 GB laptop GPU by streaming the frozen base from host RAM one decoder layer at a time. Apache 2.0 licensed.

Open-source evaluation and testing library for LLM agents. Red teaming, vulnerability scanning, RAG evaluation, and safety testing with modular architecture. Apache 2.0 licensed.

De-facto standard for generative model evaluation.

Framework for large language model evaluations from the UK AI Security Institute.

Evaluation toolkit for LLMs across multiple backends with reusable tasks, metrics, and result tracking.

Open-source agent engineering platform with prompt management, evaluations, and optimization. Features prompt playground, LLM-as-judge evals, and GEPA prompt optimizer for production LLM features. LGPL-3.0 licensed.

Test suite for decision-model servers that implement the Jev API, reporting accuracy and probability calibration and probing prompt injection, option-order sensitivity, irrelevant padding and repeated-call consistency. Apache 2.0 licensed.

Zero-dependency AI agent production-readiness toolkit with GitHub Actions score and risk-profile gates, prompt-injection fixtures, machine-readable results, and reproducible LangGraph traces. MIT licensed.

Open-source self-hostable end-to-end agent engineering and optimization platform that unifies tracing, evals, simulations, datasets, gateway, and guardrails. Built for shipping self-improving AI agents with one feedback loop from prototype to production. Apache 2.0 licensed.

Open-source evaluation library for LLM and agent applications. Built by LangChain with pre-built evaluators for common use cases including RAG, agents, and structured output validation. MIT licensed.

Agentic RL Training at Scale from Prime Intellect. Framework for large-scale reinforcement learning capable of scaling to 1000+ GPUs with fully asynchronous RL, FSDP2 training, and vLLM inference. Apache 2.0 licensed.

Curated list of artificial intelligence courses, books, video lectures, and papers for developers and researchers. MIT licensed.

Agent-auditing framework that runs adversarial and operational inspections against AI agents, producing scored JSON and Markdown reports for CLI, CI, and agent-plugin workflows. Apache-2.0 licensed.

Native inference engine optimized for DeepSeek V4 and GLM models with Metal, CUDA, and ROCm support.

The "Pytest for LLMs".

Making AI for robotics more accessible with end-to-end learning. State-of-the-art approaches for imitation learning and reinforcement learning with pretrained models, datasets, and simulated environments. Apache 2.0 licensed.

Standardized evaluation metrics.