api7/aisix
Self-hosted Apache-2.0-licensed AI gateway written in Rust, with OpenAI-compatible endpoints, an Anthropic-compatible Messages endpoint, model routing, traffic policies, Prometheus metrics, and OTLP trace export.
EricLBuehler/mistral.rs
Fast, flexible Rust-native LLM inference engine built on Candle. Supports text, vision, audio, image generation, and embeddings with hardware-aware auto-tuning.
FlashML-org/FreeToken
Edge-native Mixture-of-Experts serving engine for running frontier-scale open-weight models across consumer GPUs, CPUs, and host memory, with OpenAI- and Anthropic-compatible APIs. Apache-2.0 licensed.
oobabooga/text-generation-webui
Web UI for running local LLMs with multiple backends, extensions, and model formats.
VibePod/vibepod-cli
Unified CLI for running Claude Code, Codex, OpenCode, and other coding agents in isolated Docker or Podman containers, with local HTTP traffic and token metrics plus a side-by-side comparison dashboard. MIT licensed.
JustVugg/colibri
Zero-dependency C inference engine that runs large Mixture-of-Experts models locally by streaming experts across disk, RAM, and VRAM.