optillm

Optimizing inference proxy for LLMs with load balancing, failover, and request routing across multiple providers and models. Improves reliability and performance for production deployments. Apache 2.0 licensed.

View on GitHub Unknown

Guide pending

We haven't written this guide yet.

Guides are drafted from the project's own README and metadata. Until one exists, check the product's repository and documentation for authoritative information.

Nearby on the map

Related projects

AirLLM

lyogavin/airllm

Single-GPU 70B inference stack with strong memory/performance optimizations for local deployment on commodity hardware.

unknown unknownactivity unknown

Mirage Persistent Kernel

mirage-project/mirage

Compiler that fuses model execution into a single mega-kernel for tighter performance.

unknown unknownactivity unknown

vLLM-Ascend

vllm-project/vllm-ascend

Hardware plugin for running vLLM on Huawei Ascend accelerators.

unknown unknownactivity unknown

CTranslate2

OpenNMT/CTranslate2

Fast inference engine for Transformer models supporting OpenNMT and Hugging Face models. Optimized for CPU and GPU with batching, quantization (INT8/FP16), and dynamic memory management. Powers faster-whisper and other production deployments. MIT licensed.

unknown unknownactivity unknown

llama-swap

mostlygeek/llama-swap

Intelligent model swapping proxy for llama.cpp. Enables seamless hot-swapping between different GGUF models without restarting the server, with automatic model loading/unloading and OpenAI-compatible API. MIT licensed.

unknown unknownactivity unknown

Fugusashi

eulogik/fugusashi

Open-source intelligent model router with CMA-ES evolved routing weights, federated learning for privacy-preserving collaborative routing, and human-interpretable explanations for every decision. OpenAI-compatible API with web dashboard. MIT licensed.

unknown unknownactivity unknown