• Sources: primary, discussion
  • Summary: Amazon engineers moved internal ticket labeling from Claude Sonnet 4.6 on Bedrock to the Gemma 4 E4B build of Strands Decider on SageMaker ml.g5.xlarge instances. The decision model returns probabilities for a fixed set of options instead of generating JSON. Average call time fell from 4,215 ms to 897 ms, tokens per ticket fell from 6,724 to 2,010, and cost per ticket fell from about $0.0222 to about $0.0005. They rolled the change out with an AppConfig switch that could run both models in parallel, and found that the order of JSON keys in the input shifted the output probabilities by up to 0.19.
  • Why it matters: High-volume classification steps in agent pipelines can often move from a frontier LLM to a small decision model, provided you shadow-run both on live traffic before switching.
  • Background: A decision model is a classifier built from an LLM that scores a fixed list of options rather than generating free text, so each call produces one output token. Running both systems on the same traffic before a switch, called shadow mode, is a standard way to compare behavior on real inputs.
  • Comments: The Hacker News thread reached 180 points and 60 comments.