Skip to content- Sources: primary, discussion
- Summary: DigitalOcean serves the roughly 2.78 trillion parameter Kimi K3 on 8x NVIDIA HGX B300 and AMD MI350X servers with llm-d and vLLM, using MXFP4 weights to leave KV cache headroom. Moonshot's Kimi Vendor Verifier first returned 30 failures, 26 of them because the proxy passed per-message dynamic tools through to vLLM unvalidated, and further fixes covered strict-mode tool schemas under tool_choice auto, thinking parameter mapping, and four OpenAI streaming spec deviations consolidated into one ordered pipeline. A 67 to 70 token prompt usage drift from injected system messages was also fixed.
- Why it matters: Anyone self-hosting open weights behind an OpenAI-compatible gateway should expect model-specific API extensions and stream quirks to silently lower quality unless checked against the vendor's own verification suite.
- Background: Open-weight labs publish weights, but each provider must reproduce the chat template, decoding parameters and tool-call parsing, and small mismatches lower benchmark scores. Moonshot released the Kimi Vendor Verifier alongside K2.6 after seeing inconsistent results from vendors serving identical weights. Schema-constrained decoding forces generated tool arguments to match a JSON Schema, which vLLM applies only when the schema has the expected strict shape.