Skip to content- Sources: primary
- Summary: The speakers propose an admission layer between the gateway and engines such as vLLM or SGLang. It returns 200 to admit, 503 with Retry-After to defer, 429 when a client exceeds its share, or 529 when every replica is full. The decision is based on request size, tenant, agent session, deadline, KV cache use and queue age. They note that admitting each request separately can leave an agent session half-finished, and they recommend scoring goodput against all arriving requests, including rejected ones. On Qwen 2.5 0.5B on an A10, a policy allowing at most two waiting requests admitted 63% and cut P95 time to first token from about 2.5 s to about 1 s, while KV cache use peaked at 0.5% and a KV-threshold policy would have admitted everything.
- Why it matters: Teams hosting models for coding agents should shed load at the ingress based on signals that change within seconds, and agents should honor Retry-After with jitter.
- Background: During decode, each admitted LLM request holds a growing KV cache in GPU memory until it finishes or the client disconnects, so admitting a request commits memory for its whole life. Gateway rate limits count requests and cannot see that memory cost, while the engine finds out only after it has accepted the request.