Monitoring, Evaluation & Observability

#1 open-source LLM observability platform.

AI observability & evaluation platform.

ML & LLM monitoring framework.

Production-ready LLM evaluation platform.

AI Gateway to call 100+ LLM APIs in OpenAI format with unified cost tracking, guardrails, load balancing, and logging.

OpenTelemetry-native LLM observability platform with GPU monitoring, evaluations, prompt management, and guardrails.

Runtime-evidence toolkit for agents that coordinates PyTorch Profiler and Nsight Systems captures, preserves native traces, and compares GPU-kernel and inference runs. MIT licensed.

Open-source observability for GenAI/LLM applications based on OpenTelemetry with 25+ integration backends.

Open-source LLMOps platform combining prompt playground, prompt management, LLM evaluation, and observability.

Open-source agent engineering platform with prompt management, evaluations, and optimization. Features prompt playground, LLM-as-judge evals, and GEPA prompt optimizer for production LLM features. LGPL-3.0 licensed.

Open-source LLM observability platform with request logging, caching, prompt management, rate limiting, and cost analytics. Apache 2.0 licensed.

Open-source evaluation and testing library for LLM agents. Red teaming, vulnerability scanning, RAG evaluation, and safety testing with modular architecture. Apache 2.0 licensed.

Zero-dependency AI agent production-readiness toolkit with GitHub Actions score and risk-profile gates, prompt-injection fixtures, machine-readable results, and reproducible LangGraph traces. MIT licensed.

Blazing fast AI Gateway to route 200+ LLMs with unified API. Integrated guardrails, load balancing, fallbacks, and cost tracking. MIT licensed.

Manages unified access to generative AI services built on Envoy Gateway. Kubernetes-native AI gateway for routing, load balancing, and managing LLM traffic with enterprise-grade reliability. Apache 2.0 licensed.

Self-hosted Apache-2.0 AI gateway and MCP server: per-key virtual tokens with token budgets, exact and lexical-approximate response caching (semantic-grade matching requires an attached embedding endpoint), an append-only audit chain, deterministic local prompt enhancement that makes no provider call, and reverse MCP governance that exposes upstream MCP servers and OpenAPI 3 operations as allow-listed tools.

Cloud-native LLMOps platform with prompt management, versioning, and observability. Features collaborative prompt editing, A/B testing, and cost analytics. Apache 2.0 licensed.

Comprehensive suite for LLM-based AI app development from prototyping to production. Includes prompt engineering, evaluation, and deployment tools with VS Code integration. MIT licensed.

Visual programming environment for battle-testing prompts and evaluating LLM outputs. Features node-based prompt chains, multi-model comparison, and hypothesis testing. MIT licensed.

Open-source self-hostable end-to-end agent engineering and optimization platform that unifies tracing, evals, simulations, datasets, gateway, and guardrails. Built for shipping self-improving AI agents with one feedback loop from prototype to production. Apache 2.0 licensed.

AI-powered multi-cluster Kubernetes dashboard with GPU workload monitoring, AI pipeline observability, and CNCF ecosystem integrations. Apache 2.0 licensed.

Records a coding-agent run below the harness and replays it offline with the network off, or forks it from a checkpoint onto another model.

CLI that health-checks NVIDIA GPU nodes against reference values for the same model (power limit, sustained clocks, BF16 TFLOPS, memory bandwidth, throttle reasons, PCIe, ECC), with a per-minute watch mode for rentals. MIT licensed.