Mesh LLM runs distributed inference over iroh

Category: AI
Status: discussion
Sources: iroh blog, HN discussion
Summary: Published 2026-07-11, Mesh LLM pools GPUs and memory across machines and exposes them as a single OpenAI-compatible endpoint. It uses iroh for authenticated, NAT-traversing QUIC connections between nodes (hole-punching and relay fallback, no central server), with three routing strategies: local execution, routing to a peer already running the model, or splitting a large model across machines in a pipeline-parallel "Skippy" mode where layer ranges become stages and activations flow sequentially. The protocol negotiates connection types over QUIC ALPN and demultiplexes streams with single-byte tags. The code is on GitHub.
Why it matters: It is a concrete pattern for running models too large for one machine across commodity peers without a coordinator, relevant to local-inference and self-hosted setups.
Follow-up: Watch for benchmarks of the pipeline-parallel path, fault handling when a stage node drops, and whether it moves past an early project.