Policy Loopholes in Agent Evaluation: When Policy Ambiguity Masquerades as Agent Error
Hongliu Cao (Amadeus) audits two tau2-bench domains and finds that ambiguous, silent or contradictory natural-language policies admit several defensible actions, which a single gold trajectory scores as agent errors.
Ask this paper
Audit method: A two-stage audit combines proactive clause analysis of the policy document with reactive verification against agent traces. It finds 17 loopholes, 12 surfaced by clause analysis alone and 5 confirmed by both stages, sorted into a five-category taxonomy.
Effect on scores: Tasks affected by loopholes lower scores in model-specific ways and make every model less consistent across repeated trials.
Two-factor mechanism: Both domains contain policy gaps, but only airline scores drop. Loopholes become exploitable when policy complexity exceeds what the tools can enforce, so agents resolve the gaps inconsistently.
Fix at the policy level: Per-task patches leave the ambiguous clause in place and can produce contradictory gold answers, while policy-level corrections resolve the ambiguity once for every affected task.
Abstract
Agent benchmarks evaluate policy compliance but assume each policy determines a unique correct action. Natural-language policies can violate this assumption through silence, ambiguity, or contradiction, admitting multiple defensible readings that a single gold trajectory cannot capture. Auditing two $\tau^2$-bench domains, we develop a taxonomy of such policy loopholes and show that affected tasks produce unreliable scores: they lower scores across different models in different ways and make every model less consistent across repeated trials. A cross-domain comparison reveals that exploitability requires both policy ambiguity and tool permissiveness: when policy complexity exceeds what tools can enforce, agents resolve gaps inconsistently and scores become unreliable. Policy specification quality sets the ceiling on evaluation quality. Benchmark developers should audit policies before collecting gold annotations.