Results for "inference"

Distributed LLM inference connecting home devices into a powerful cluster. More devices means faster inference via tensor parallelism over Ethernet. Supports Linux, macOS, Windows, ARM, and x86_64 AVX2 CPUs. MIT licensed.

Toolkit for compressing, deploying, and serving LLMs from OpenMMLab. 4-bit inference with 2.4x higher performance than FP16, distributed multi-model serving across machines.

Lightweight, standalone C++ inference engine for Google's Gemma models. Optimized for on-device deployment with minimal dependencies and efficient memory usage. Apache 2.0 licensed.

Cost-efficient and pluggable infrastructure components for GenAI inference. Kubernetes-native control plane for vLLM with distributed KV cache, heterogeneous GPU serving, and intelligent routing. Apache 2.0 licensed.

Kernel library for LLM serving. High-performance CUDA kernels for attention, sampling, and matrix multiplication. Powers vLLM, SGLang, and other inference engines with optimized GPU kernels. Apache 2.0 licensed.

Pure C/C++ inference engine with GGUF format support. The gold standard for CPU/GPU/Apple Silicon on-device running. Includes llama-server for OpenAI-compatible API. Now at 100K+ stars.

Flexible framework for heterogeneous CPU-GPU LLM inference and fine-tuning. Enables running large MoE models by offloading experts to CPU with BF16/FP8 precision support.

Official inference framework for 1-bit LLMs (BitNet b1.58). Enables running large models on CPU with minimal memory footprint. Features custom kernels for ternary weight quantization and efficient matmul operations. MIT licensed.

NVIDIA's official high-performance inference backend.

NVIDIA's production-grade open-source inference serving software. Supports multiple frameworks (TensorRT, PyTorch, ONNX) with optimized cloud and edge deployment.

Python-free Rust inference server with OpenAI API compatibility. Supports GGUF and SafeTensors formats with hot model swap, auto-discovery, and single binary deployment for zero-dependency inference. Apache 2.0 licensed.

Compiler that fuses model execution into a single mega-kernel for tighter performance.

Evaluation platform for benchmarking language and multimodal models across large benchmark suites.

High-speed LLM inference for local deployment on consumer GPUs. Achieves up to 11x speedup over llama.cpp on RTX 4090 by exploiting power-law neuron activation patterns. MIT licensed.

Optimizing inference proxy for LLMs with load balancing, failover, and request routing across multiple providers and models. Improves reliability and performance for production deployments. Apache 2.0 licensed.

High-performance inference engine optimized for Chinese AI accelerators (Cambricon MLU, Hygon DCU, Huawei Ascend). Features service-engine decoupled architecture with elastic scheduling, PD disaggregation, and global KV cache management. Powers JD.com's core retail businesses. Apache 2.0 licensed.

Next-gen serving framework with RadixAttention. Powers xAI's production workloads at 100K+ GPUs scale.

State-of-the-art serving engine with PagedAttention and continuous batching. Currently the fastest production-grade LLM server.

Fast and lightweight multimodal LLM inference engine for mobile and edge devices. Optimized for running vision-language models on resource-constrained hardware with efficient memory management. MIT licensed.

Attention-free language model architecture with linear-time inference, training code, inference examples, and an active open-source ecosystem.

Production-grade serving platform for Kimi (Moonshot AI). Features distributed KV cache pool with intelligent offloading, prefill/decode disaggregation, and cross-instance KV reuse. Integrated with vLLM, SGLang, and TensorRT-LLM. Apache 2.0 licensed.

High-performance neural network inference framework optimized for mobile platforms. No third-party dependencies, cross-platform, and runs faster than all known open-source frameworks on mobile CPU. Powers Tencent apps including QQ, WeChat, and Pitu. BSD-3-Clause licensed.

vLLM fork optimized for role-play and creative writing. Supports extensive quantization methods (AQLM, AWQ, GPTQ, GGUF, FP8) and modern samplers. Active development with multi-LoRA and speculative decoding support.

High-performance inference and training for ONNX models across hardware.