Self-hosted inference orchestrators compared: LocalAI, exo, GPUStack, vLLM — SkimNews
Get the Tech newsletter
Daily tech — startups, AI labs, chips, the launches that shape the next decade. Free.
- LocalAI gained a distributed mode in June 2026 via --p2p, with libp2p/EdgeVPN discovery and a NATS-based "v3" router aware of VRAM and prefix caches; backend images are cosign-signed and the survey notes raw LLM throughput trails a dedicated engine by "tens of percent."
- exo achieves 3.2× scaling on four Apple Silicon devices using MLX and RDMA over Thunderbolt 5, but as of September 2026 remains CPU-only on Linux with NVIDIA and AMD support listed as "under development."
- GPUStack supports nine accelerator vendors including Ascend, Hygon, and MThreads, and is described in the survey as "what I see actually deployed as clusters in Asia" — with users, API keys, metering, Prometheus/Grafana, and automatic recovery of failed models.
- Ollama paired with Open WebUI is positioned as the simplest single-machine answer — one binary, one model at a time — with no native cluster story beyond Open WebUI round-robining several Ollama URLs.
- NVIDIA Dynamo and llm-d target rack-scale problems with disaggregated prefill/decode, KV-cache-aware routing, and multi-tier KV storage on Kubernetes; the survey calls them "the wrong tool below a rack."
- CoderAI, flagged by the author as their own project, uniquely offers three-tier escalation — local card to owned machine to RunPod rented GPU with a price cap — plus LoRA/QLoRA training data-parallel across nodes from the same server.
Why it matters: For GPU-equipped engineers, picking the wrong orchestrator means missing cluster mode, KV-aware routing, or escalation. Hardware now drives the choice: Ollama for one machine, exo for Apple Silicon stacks (3.2× over Thunderbolt 5), LocalAI for breadth, GPUStack for ops dashboards, NVIDIA Dynamo for rack-scale.
Ask SkimNews
