FreeToken
Edge-native Mixture-of-Experts serving engine for running frontier-scale open-weight models across consumer GPUs, CPUs, and host memory, with OpenAI- and Anthropic-compatible APIs. Apache-2.0 licensed.
An Apache-2.0 serving engine that aims to run large open-weight MoE models on consumer GPUs, CPUs, and host memory.
What it is
FreeToken is an edge-native Mixture-of-Experts serving engine for running frontier-scale open-weight models on personal hardware. Rather than assuming a datacenter, it treats the GPUs, CPUs, host memory, and interconnects in a single machine as one elastic inference platform. It exposes OpenAI- and Anthropic-compatible APIs, so tools built for those interfaces can point at a local server. The project is licensed under Apache-2.0 and is listed in the High-performance Serving and API Servers category.
Key features
- Bandwidth-adaptive CPU and GPU co-execution, plus double-buffered prefill streaming and a global LRU expert cache.
- Semantic anchor checkpoints for recurrent state and KV caches, so edits like tool calls or thinking blocks avoid recomputing context.
- Runtime re-allocation of VRAM between expert caches and KV memory, without restarting the engine or reloading weights.
- Support for several open-weight MoE models, including DeepSeek-V4-Flash, Qwen3.6-35B-A3B, and GLM-5.2, across formats such as MXFP4, NVFP4, FP8, and BF16.
- Compatibility with OpenAI and Anthropic style APIs, for use with coding and tool-calling agents.
Use cases
- Running a large MoE model on a gaming PC or workstation that cannot hold the full model in VRAM.
- Pointing a coding agent or tool-calling client at a local endpoint instead of a hosted API.
- Keeping long agentic sessions responsive by reusing cached context across tool calls.
- Experimenting with frontier open-weight models on laptops or desktops with NVIDIA RTX 30, 40, or 50 series GPUs.
Who it's for
Developers and researchers who want to run large open-weight MoE models on hardware they already own. It suits people building local coding agents or experimenting with memory-constrained inference.
Getting started
The README offers a desktop app for Windows and Linux, downloadable from flashml.ai, which sets up the engine and provides a GUI. For the command-line route, it describes installing with uv or pip and building from source, with links to install, quick start, supported models, and CLI reference docs. The exact install commands are in the README's Getting Started section and linked documentation, so check those directly before running anything.
Alternatives
The directory lists related entries such as vLLM, SGLang, LMDeploy, and TensorRT-LLM, which are serving projects in the same category. These are provided as related entries only, and this guide makes no claim about how they compare.
Drafted with AI from the project's public documentation; verify against the source. Written 2 Nov 2026.