High-performance Serving & API Servers

State-of-the-art serving engine with PagedAttention and continuous batching. Currently the fastest production-grade LLM server.

Kubernetes-native distributed LLM inference framework. Donated to CNCF by RedHat, Google, and IBM. Intelligent scheduling, KV-cache optimization, and state-of-the-art performance across accelerators.

Toolkit for compressing, deploying, and serving LLMs from OpenMMLab. 4-bit inference with 2.4x higher performance than FP16, distributed multi-model serving across machines.

LLM inference engine for agentic workloads with a C++ scheduling control plane, Python execution plane, modular kernels, and KV-cache lifecycle management. MIT licensed.

Multi-modal inference stack extending vLLM for image, audio, and video generation workloads with a unified serving interface. Apache 2.0 licensed.

Supercharge LLM inference with the fastest KV Cache layer. 3-10x delay savings and GPU cycle reduction for multi-round QA and RAG. Integrates seamlessly with vLLM for distributed, high-throughput deployments. Apache 2.0 licensed.

Kubernetes-native production stack for vLLM inference. Automated deployment, autoscaling, and monitoring for enterprise-grade LLM serving. Built by the vLLM team for seamless integration.

Kubernetes operator for LLM serving with GPU scheduling and model lifecycle management across vLLM, SGLang, and TensorRT-LLM.

Next-gen serving framework with RadixAttention. Powers xAI's production workloads at 100K+ GPUs scale.

NVIDIA's official high-performance inference backend.

vLLM fork optimized for role-play and creative writing. Supports extensive quantization methods (AQLM, AWQ, GPTQ, GGUF, FP8) and modern samplers. Active development with multi-LoRA and speculative decoding support.

Cost-efficient and pluggable infrastructure components for GenAI inference. Kubernetes-native control plane for vLLM with distributed KV cache, heterogeneous GPU serving, and intelligent routing. Apache 2.0 licensed.

Self-hosted Apache-2.0-licensed AI gateway written in Rust, with OpenAI-compatible endpoints, an Anthropic-compatible Messages endpoint, model routing, traffic policies, Prometheus metrics, and OTLP trace export.

NVIDIA's production-grade open-source inference serving software. Supports multiple frameworks (TensorRT, PyTorch, ONNX) with optimized cloud and edge deployment.

Fast, flexible Rust-native LLM inference engine built on Candle. Supports text, vision, audio, image generation, and embeddings with hardware-aware auto-tuning.

Flexible framework for heterogeneous CPU-GPU LLM inference and fine-tuning. Enables running large MoE models by offloading experts to CPU with BF16/FP8 precision support.

Mozilla's single-file distributable LLM solution. Bundle model weights, inference engine, and runtime into one portable executable that runs on six OSes without installation.

Unified, production-ready inference API for LLMs, speech, and multimodal models. Drop-in GPT replacement with single-line code changes. Supports thousands of models with auto-batching and distributed inference.

Alibaba's high-performance LLM inference acceleration engine. Powers production LLM services across Taobao, Tmall, and Alibaba's international AI platform. Supports PagedAttention, FlashAttention, FlashDecoding, INT8/INT4 quantization, and heterogeneous hardware (GPU/ARM CPU/Intel). Apache 2.0 licensed.

Minimal Python framework for building custom AI inference servers with full control over logic, batching, and scaling. 2x faster than FastAPI with built-in batching, streaming, and multi-GPU autoscaling. Apache 2.0 licensed.

Pure Python-based LLM inference and serving framework with lightweight design, easy extensibility, and high-speed performance. Integrates optimizations from FasterTransformer, TGI, vLLM, and SGLang.

FastAPI-based API server for ExLlamaV2/V3 backends. OpenAI-compatible API with support for model loading/unloading, embeddings, speculative decoding, multi-LoRA, and streaming.

GPU cluster manager that orchestrates inference engines like vLLM and SGLang. Automated engine selection, parameter optimization, and distributed multi-GPU deployment for high-performance AI workloads.

Production-grade platform for running any open-source LLMs as OpenAI-compatible API endpoints. Supports 50+ models with built-in streaming, batching, and auto-acceleration. Apache 2.0 licensed.