Inference Engines & Serving

Dead-simple local LLM runner with a one-line install, model registry, and OpenAI-compatible API.

Pure C/C++ inference engine with GGUF format support. The gold standard for CPU/GPU/Apple Silicon on-device running. Includes llama-server for OpenAI-compatible API. Now at 100K+ stars.

State-of-the-art serving engine with PagedAttention and continuous batching. Currently the fastest production-grade LLM server.

Open-source on-device AI platform covering discovery, model running, sandboxed execution, and evaluation of open models.

Linux distribution for fully local AI inference on Raspberry Pi 5 and 4, optimized for running open models at the edge.

Deployment engine that compiles and runs LLMs across browsers, mobile devices, and local hardware.

High-performance in-browser LLM inference engine. Runs models directly in the browser with WebGPU acceleration.

Official Python bindings for llama.cpp.

User-friendly llama.cpp fork focused on role-playing and creative writing.

Container-centric tool for simplifying local AI model serving. Automatically detects GPUs, pulls optimized container images, and runs models securely in rootless containers with enterprise-grade isolation.

Google's production-ready on-device ML and GenAI deployment framework. Supports Android, iOS, Web, Desktop, and IoT targets with GPU/NPU acceleration via a unified edge-first runtime. Apache 2.0 licensed.

Production-ready runtime for deploying LLMs on edge devices with low-latency inference and optimized hardware paths for mobile and embedded platforms.

Run frontier AI locally by connecting all your devices into an AI cluster. Features automatic device discovery, RDMA over Thunderbolt for 99% latency reduction, topology-aware auto parallel, and tensor parallelism. Uses MLX backend for distributed inference across Apple Silicon devices. Apache 2.0 licensed.

Native inference engine optimized for DeepSeek V4 and GLM models with Metal, CUDA, and ROCm support.

Native C implementation of Qwen3.8-Flash-Next for local CPU inference, with terminal chat, a resident OpenAI-compatible API, and reproducible benchmarks.

Apple-centric inference server for local-first AI workflows with model management, GPU orchestration, and OpenAI-compatible APIs for self-hosted deployment. Apache 2.0 licensed.

Terminal tool and TUI that right-sizes LLM models to hardware specs and scores local compatibility across GPU, CPU, and RAM. MIT licensed.

Compact 45M-parameter foundation model and 14MB inference engine for tool calling and structured extraction on tiny devices. MIT licensed.

Zero-dependency C inference engine that runs large Mixture-of-Experts models locally by streaming experts across disk, RAM, and VRAM.

MLX-native server that speaks the Anthropic Messages API so the unmodified Claude Code CLI runs against local models on Apple Silicon, with parsing for local models' tool-call formats. MIT licensed.

Pure MLX runtime for the vision and audio towers of NVIDIA Nemotron 3 Nano Omni on Apple Silicon, tested for parity against NVIDIA's PyTorch reference. MIT licensed.

Hardware-aware local inference engine that profiles host hardware, recommends suitable open models, and tunes execution across Apple Silicon, NVIDIA, AMD, and CPU. Apache 2.0 licensed.

Dependency-free TypeScript core that estimates inference memory (weights, KV cache, overhead) of any Hugging Face model from its config.json and safetensors metadata, covering MoE, MLA, sliding-window and linear-attention layers; runs in the browser at [modelvram.com](https://modelvram.com/llm-vram-calculator/). MIT licensed.

Ollama-compatible LLM server on a llama.cpp fork, tuned for speculative decoding on NVIDIA GPUs. Keeps Ollama's commands, API and model store, drafts with the model's own MTP head, and can uncensor a model with a control vector applied at load. MIT licensed.