Talk reports frontier models doing the reconnaissance but missing the logic leap in an access-control exploit

Sources: watch
Channel: AI Engineer (2026-07-24, 1,210 views, 5.0 over 34 ratings)
Summary: Uri Rolls of Arithmetic and Hugging Face cofounder Thom Wolf describe a target environment chaining Keycloak, Vault, and a broker, entered as a low-privileged user, that contains a real access-control flaw: one check validates the administrator by name while another checks by ID, so a user who renames themselves to the administrator inherits the privilege. They report that GPT-5.5 and Opus probe the environment thoroughly and reach the check but do not make the inference. Their proposal is to build cyber training data by having human vulnerability researchers find zero-days in open-source software, then wrapping each in a black-box environment where discovery and exploitation steps are deterministically graded. They report exactly one solve at k=1 on the resulting access-control benchmark, and argue open models good at this class of reasoning would give defenders a durable edge.
Why it matters: It puts a measured boundary on where current models stop in an exploitation chain, which is the number missing from most claims that models can or cannot find real vulnerabilities.