Tracing Agentic Failure
Free while signed in. Answers cite the passages they came from.
Finding which step in a failed agent run actually caused the failure usually means either labeling failure data or running expensive per-step prompting. This Microsoft and UW-Madison work skips both by learning what success looks like and flagging deviations from it.
Train on success, judge failure: OAT uses one-class learning with neural controlled differential equations to model the latent dynamics of successful trajectories, then scores each step of a failed run by how far it strays from that learned flow.
Cheap and label-free: With only 100 successful trajectories and no failure labels, it turns failure attribution into anomaly detection, avoiding the annotation and prompting costs that make current methods impractical at scale.
Strong, fast results: It delivers a 200 to 5000 times speedup over prompting-based attribution while improving F1 by 20% in-domain and 7% out-of-distribution.
Why it matters: Production agents fail in long, probabilistic, tool-mediated runs where the decisive misstep is hard to localize, and a cheap detector that only needs success data makes routine debugging feasible.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack