vLLM-Ascend

Single-GPU 70B inference stack with strong memory/performance optimizations for local deployment on commodity hardware.

Compiler that fuses model execution into a single mega-kernel for tighter performance.

Fast inference engine for Transformer models supporting OpenNMT and Hugging Face models. Optimized for CPU and GPU with batching, quantization (INT8/FP16), and dynamic memory management. Powers faster-whisper and other production deployments. MIT licensed.

Intelligent model swapping proxy for llama.cpp. Enables seamless hot-swapping between different GGUF models without restarting the server, with automatic model loading/unloading and OpenAI-compatible API. MIT licensed.

Optimizing inference proxy for LLMs with load balancing, failover, and request routing across multiple providers and models. Improves reliability and performance for production deployments. Apache 2.0 licensed.

Open-source intelligent model router with CMA-ES evolved routing weights, federated learning for privacy-preserving collaborative routing, and human-interpretable explanations for every decision. OpenAI-compatible API with web dashboard. MIT licensed.