Skip to content- Sources: primary
- Summary: Cartesia trained a 2-stage byte H-Net with 3.67B parameters, a 40-layer Transformer core, and Mamba-2 outer stages at matched FLOPs per byte. It reached SmolLM3's final average score after about 1T tokens against about 11T. On a scaling ladder, a 1-stage H-Net needed 22% less compute than BPE for equal validation loss, with a 3.10x median gain on out-of-distribution code languages.
- Why it matters: Tokenizer-free models now show a scaling result at the 3B size, which matters for code and multilingual workloads that BPE serves poorly.