Black-box audit detects when a gateway swaps or dilutes the model you asked for

Sources: arXiv 2607.20860
Summary: Yuewei Zhang, Zhi-Hai Zhang, and Hanzhang Qin submitted a preprint on 2026-07-23 describing IRIS, a text-only auditing method that checks whether a commercial LLM gateway serves the model it advertises, either substituting a cheaper backend outright or routing only a fraction of requests to the promised one. It prompts for random number and string generation to fingerprint the backend, then sizes its own query budget from a cheap pilot phase that fits an exponential query-error decay curve. The authors report 0.99 AUROC separating backends within the Qwen3 family, detection of a 0.3 dilution rate at 0.85 mean power with a 0.017 false-positive rate across OpenRouter, recovery of routing fractions within 0.04, and flagging of 14 of 15 same-model provider pairs through quantization differences. The figures are the authors' own and not independently reproduced.
Why it matters: Teams routing production traffic through a gateway currently take the advertised model on trust, and this puts a measurable query budget on checking it.
Follow-up: Watch for independent runs against named gateways and for any provider response on routing disclosure.