Mesh LLM pools idle GPUs into one OpenAI-compatible API
Get the Tech newsletter
Daily tech — startups, AI labs, chips, the launches that shape the next decade. Free.
- Mesh LLM pools GPUs and memory across as many machines as a user adds and exposes the cluster as one OpenAI-compatible API at localhost:9337/v1, letting existing OpenAI clients point at it with no code changes.
- The catalog ships with 40+ models ranging from ~500-million-parameter models that fit on a laptop to 235B mixture-of-experts giants, covering the full size curve the source highlights.
- A split mode internally called "Skippy" partitions oversized models by layer ranges — e.g., layers 0-15 on one node, 16-31 on the next — and streams activations down a pipeline so several modest machines can run a model none could hold alone.
- Every node boots an iroh endpoint as its identity (a public key), and iroh handles the NAT traversal, hole-punching, and authenticated QUIC connections between any two machines — there is no central server in the mesh.
- The protocol rides on QUIC's ALPN negotiation with three protocol IDs, and Mesh LLM runs two iroh relays in different regions as fallback paths for nodes that cannot reach each other directly.
- The installer is roughly 18 MB; a mobile app built on iroh's Swift SDK is planned, intended to speak the ACP agent standard so other clients can join the mesh.
Why it matters: Teams sitting on idle GPUs in offices, closets, or under desks gain a way to run models too large for any single box while keeping inference on hardware they own, with no metered API bill. The OpenAI-compatible localhost endpoint means the swap is drop-in for existing clients, removing the usual migration tax for moving off a large provider.
