Ollama Herd

Self-hosted router that spreads LLM, embedding, image and speech-to-text requests across machines running Ollama or mlx_lm.server, choosing a node per request by memory fit, thermal state, queue depth and loaded models. MIT licensed.

View on GitHub Unknown

Guide pending

We haven't written this guide yet.

Guides are drafted from the project's own README and metadata. Until one exists, check the product's repository and documentation for authoritative information.

Nearby on the map

Related projects

vLLM

vllm-project/vllm

State-of-the-art serving engine with PagedAttention and continuous batching. Currently the fastest production-grade LLM server.

96k 24kPythonpushed 2 hours ago

llm-d

llm-d/llm-d

Kubernetes-native distributed LLM inference framework. Donated to CNCF by RedHat, Google, and IBM. Intelligent scheduling, KV-cache optimization, and state-of-the-art performance across accelerators.

unknown unknownactivity unknown

LMDeploy

InternLM/lmdeploy

Toolkit for compressing, deploying, and serving LLMs from OpenMMLab. 4-bit inference with 2.4x higher performance than FP16, distributed multi-model serving across machines.

unknown unknownactivity unknown

TokenSpeed

lightseekorg/tokenspeed

LLM inference engine for agentic workloads with a C++ scheduling control plane, Python execution plane, modular kernels, and KV-cache lifecycle management. MIT licensed.

unknown unknownactivity unknown

vLLM-Omni

vllm-project/vllm-omni

Multi-modal inference stack extending vLLM for image, audio, and video generation workloads with a unified serving interface. Apache 2.0 licensed.

unknown unknownactivity unknown

LMCache

LMCache/LMCache

Supercharge LLM inference with the fastest KV Cache layer. 3-10x delay savings and GPU cycle reduction for multi-round QA and RAG. Integrates seamlessly with vLLM for distributed, high-throughput deployments. Apache 2.0 licensed.

unknown unknownactivity unknown