• Sources: primary, discussion
  • Thread: Inkling 2 Small open weights
  • Summary: The model card describes a 48-layer decoder-only mixture of experts that routes each token to 8 of 384 experts. It mixes local and global attention layers and encodes images with a vision transformer. The model was pretrained, midtrained and post-trained in NVFP4. That checkpoint needs at least 500GB of aggregate VRAM, such as 2 B300 or 4 H200 GPUs, and the MXFP8 checkpoint needs 800GB. It runs on SGLang, vLLM, TokenSpeed and Unsloth, and can be post-trained with Miles, SkyRL or Tinker.
  • Why it matters: An Apache 2.0 model in this class fits on a single 4-GPU H200 node, and the native NVFP4 checkpoint avoids the accuracy loss of quantizing after training.
  • Background: A mixture-of-experts model holds many feed-forward blocks, called experts, and a router sends each token to only a few of them. Total parameters set memory needs, while active parameters set the compute per token. NVFP4 is NVIDIA's 4-bit floating point format with per-block scale factors, supported in hardware on Blackwell GPUs. Training natively in such a format means the published weights are the ones the model was optimized in.
  • Comments: The r/LocalLLaMA release thread centers on the hardware floor. The 500GB VRAM minimum puts the model out of reach of consumer rigs.