• Sources: primary
  • Summary: llm-d 1.0 adds multi-tenant priority and fairness flow control, router high availability with zero-downtime updates, DisaggregatedSet for prefill and decode aware rolling updates, and an async processor that pulls from Pub/Sub or Redis queues. The router builds on Gateway API Inference and routes requests to replicas that hold cached prefixes, with KV cache offload to host memory and SSD.
  • Why it matters: Teams serving their own models on Kubernetes get a stable, composable alternative to round-robin load balancing in front of vLLM or SGLang.