ToolPrivacyBench audits whether tool-using agents leak private data to the wrong tools

Category: Paper
Status: developing
Sources: arXiv 2606.28061
Summary: A preprint posted 2026-06-26 introduces ToolPrivacyBench, a 2,150-case benchmark (1,150 synthetic privacy-sensitive business workflows plus 1,000 cases adapted from existing multi-tool and function-calling benchmarks) that audits an agent's full execution trajectory rather than its final answer. After an agent runs against mock backends, an evaluator compares recorded tool arguments and backend audit logs against a per-case policy to check whether private data reached only authorized tools. Across nine agents the authors report that successful task completion does not imply appropriate disclosure: agents finish tasks while passing unnecessary private information through intermediate tool calls.
Why it matters: It formalizes a need-to-know disclosure boundary for tool-using agents, the same data-leak surface raised by the Codex file-exclusion gap in Top stories.
Follow-up: Watch for the dataset or evaluator release and independent reproduction.