Prompt injection reframed as role confusion in ICML 2026 paper

Category: Security
Status: confirmed
Sources: arXiv 2603.12277, project page, code, discussion
Summary: Charles Ye, Jasmine Cui, and Dylan Hadfield-Menell (MIT) argue in an ICML 2026 paper that prompt injection succeeds because models infer roles such as user, assistant, think, and tool from writing style rather than from the provider-applied structural tags, so attacker text that reads like a privileged role overrides the boundary. Using linear probes for internal "Userness" and "CoTness", they report a chain-of-thought forgery attack that injects fake reasoning and succeeds about 60% of the time; stripping the style ("destyling") dropped success from 61% to 10%.
Why it matters: It locates prompt-injection robustness in how models represent roles internally, which points defenses at role representation rather than only input filtering.
Follow-up: Watch for independent reproduction of the destyling result and any vendor adoption of structural role tags.