Armature publishes 5,292 valid sessions measuring which third-party services coding agents install

Sources: primary, discussion
Summary: Armature ran 16,893 sessions and kept 5,292 as valid, recording which third-party services a coding agent reaches for when asked to add a capability. In the email provider case the repository language rather than the stated requirement decided which vendor the agent picked. The validity judge and the simulated human in the loop were both Gemini 3.7 Flash, and Armature sells services for getting products picked by coding agents.
Why it matters: The repository language, not the stated requirement, decided the winner in the email-provider case, so a team letting an agent pick a vendor is accepting a choice that varies with the codebase it was asked in.