Results for "moe"

A next-generation training engine built for ultra-large MoE models with efficient QLoRA and full-parameter fine-tuning. Apache 2.0 licensed.

Flexible framework for heterogeneous CPU-GPU LLM inference and fine-tuning. Enables running large MoE models by offloading experts to CPU with BF16/FP8 precision support.

Communication library for efficient expert-parallel training/inference pipelines, reducing MoE cross-device communication overhead. Apache 2.0 licensed.

Clean, high-performance FP8 GEMM kernels with fine-grained scaling for modern inference workloads. Apache 2.0 licensed.

Edge-native Mixture-of-Experts serving engine for running frontier-scale open-weight models across consumer GPUs, CPUs, and host memory, with OpenAI- and Anthropic-compatible APIs. Apache-2.0 licensed.

High-performance llama.cpp fork with better CPU and hybrid GPU/CPU performance, SOTA quantization types, first-class Bitnet support, and improved DeepSeek performance via MLA, FlashMLA, and fused MoE operations. MIT licensed.

Microsoft's deep learning optimization library for extreme-scale training (ZeRO, offloading, MoE).

Train a 64M-parameter LLM from scratch in just 2 hours for $3. Complete from-scratch implementation covering MoE, data cleaning, pretraining, SFT, LoRA, RLHF (DPO/PPO/GRPO), tool use, and model distillation. All core algorithms implemented in pure PyTorch without high-level abstractions. Educational framework for understanding LLM internals. Apache 2.0 licensed.

Kernel library for LLM serving. High-performance CUDA kernels for attention, sampling, and matrix multiplication. Powers vLLM, SGLang, and other inference engines with optimized GPU kernels. Apache 2.0 licensed.

State-of-the-art serving engine with PagedAttention and continuous batching. Currently the fastest production-grade LLM server.

Agentic RL Training at Scale from Prime Intellect. Framework for large-scale reinforcement learning capable of scaling to 1000+ GPUs with fully asynchronous RL, FSDP2 training, and vLLM inference. Apache 2.0 licensed.

Extreme-scale training optimizations.

Dependency-free TypeScript core that estimates inference memory (weights, KV cache, overhead) of any Hugging Face model from its config.json and safetensors metadata, covering MoE, MLA, sliding-window and linear-attention layers; runs in the browser at [modelvram.com](https://modelvram.com/llm-vram-calculator/). MIT licensed.

Versatile framework for both single- and multi-modal pre-training and post-training. Model-centric distributed recipe zoo supporting text, vision, audio, and video models with unified training interface. Apache 2.0 licensed.

Zero-dependency C inference engine that runs large Mixture-of-Experts models locally by streaming experts across disk, RAM, and VRAM.

LLM post-training framework for RL Scaling from THUDM. Supports SFT and RL training with multi-turn compilation feedback, powering projects like TritonForge for automated GPU kernel generation. Apache 2.0 licensed.

Terminal tool and TUI that right-sizes LLM models to hardware specs and scores local compatibility across GPU, CPU, and RAM. MIT licensed.

Hardware plugin for running vLLM on Huawei Ascend accelerators.

Compact vision-language model family with edge-focused deployment examples and strong OCR-oriented use cases.

Unified training framework for 600+ LLMs and 300+ MLLMs with CPT/SFT/DPO/GRPO (AAAI 2025).

OpenAI open-weight model repository with inference examples, recipes, and deployment guidance.