• Sources: primary
  • Summary: Shopify scores production conversations with an LLM judge calibrated against human labels. A self-healing loop repaired 59,535 failed GraphQL conversations, 69% of those it attempted, and turned them into training data. After six loops the post-trained model scored 74.5 against 73 for the frontier model, with a median latency of 0.87 seconds against 6.50 seconds.
  • Why it matters: The post shows a measurable path for moving token-heavy subagent work from frontier APIs to self-hosted open models. The judge, the failure-repair data loop and the trace-replay load tests are all reusable.