Skip to content- Sources: primary
- Summary: Spotify served LLM podcast recommendations with vLLM, whose beam search runs outside the core GPU engine, and used SGLang for multi-item candidate scoring. With NVIDIA it added several features to TensorRT LLM between v1.2.0rc3 and v1.3.0rc20: multiple beam sequences, out-of-tree models in trtllm-serve, embedding injection for user soft tokens, and multi-item scoring. On Qwen3 1.7B this cut cost by up to 54% for beam search and up to 7% for scoring.
- Why it matters: Beam search and scoring workloads now have one inference engine option with sampler-side beam search and soft-token personalization upstream.