Skip to content- Sources: primary, discussion
- Summary: The MIT-licensed project serves Qwen3.8-Flash-Next behind OpenAI-compatible and Anthropic-compatible endpoints on localhost, splitting a model of 24,576 experts with 10 active per token so that the GPU holds the most-used experts, RAM holds all of them, the CPU works the rest, and the SSD holds a lookup table, with speculative decoding the authors put at 1.6 to 1.8x, against requirements of 12GB or more of VRAM, 32GB or more of RAM, and about 80GB of disk. Every published figure is author-measured at 2-bit or 3-bit quantization: an RTX 5070 with 12GB and 64GB of RAM writes 94 tokens per second at Q2_0, an RX 9070 XT writes 60, and the Coder variant has half the experts removed and reaches 91 percent of the full model's SWE-bench Verified score by its own authors' measurement. The Hacker News headline's claim of an RTX 4090 at 100 tokens per second does not appear in the repository, whose fastest measured row is the RTX 5070 at 94 and which gives 100 to 140 only as a projection for an RTX 3090.
- Why it matters: A 125-billion-parameter mixture-of-experts model becomes reachable on a gaming PC, which changes what local inference can host, provided the quantization caveat travels with the numbers.