• Sources: primary
  • Summary: The agent starts from a model's default vLLM recipe and uses Optuna's Bayesian optimization to search serving parameters against four regimes: short chat, long chat, RAG and batch classification. A 50-trial run of several hours on GPT-OSS-120B found a config with 17.5% more goodput tokens, and a nine-trial run on a DeepSeek R1 distill doubled throughput and roughly halved inter-token latency. Each run outputs an HTML report and an optimized YAML recipe.
  • Why it matters: Teams self-hosting open-weight models can automate serving config search against their own traffic mix instead of tuning a single setting by hand.
  • Background: Bayesian optimization fits a model of how a score varies with parameters and picks the next trial where improvement is most likely, which needs far fewer runs than grid search when each trial is an hours-long benchmark.