• Sources: primary, discussion
  • Summary: NVIDIA's guide serves the GLM-5.3 NVFP4 mixture-of-experts model, 753B parameters with 40B active, whose roughly 464 GB checkpoint exceeds the 252 GB of HBM3e on one GB300, by running vLLM 0.30.0 with tensor parallelism of 2 across two DGX Stations linked by 400 Gb/s ConnectX-8 RDMA. The launch enables MTP speculative decoding with four draft tokens, an FP8 KV cache, prefix caching and a 98,304-token context. AIPerf on SPEED-Bench coding prompts shows output throughput peaking at 607.8 tokens per second at concurrency 16 and falling to 385.0 at concurrency 32, with mean request latency of 453.5 seconds.
  • Why it matters: It gives a pinned, reproducible recipe and a measured concurrency ceiling for running a frontier-size open-weight model on two workstations instead of a hosted API.
  • Background: Tensor parallelism splits each weight matrix across GPUs, so every token needs both machines to exchange partial results, which is why the guide checks jumbo frames, RDMA and NCCL settings before launch. A mixture-of-experts model activates only a few experts per token, but all experts must stay in memory because routing changes per token. Multi-token prediction lets a small extra head draft several tokens that the full model verifies in one step, raising throughput when drafts are accepted.
  • Comments: The Hacker News thread drew 86 points and 46 comments.