FinalityBench: An Effect-Level Benchmark for Agent Decisions Under Delayed and Conflicting Financial Finality
Abhishek Sharma builds an executable benchmark for agents resolving payment exceptions when a merchant's processor, ledger, ERP and bank feed hold contradictory beliefs about the same order, and grades on executed monetary effects rather than answer accuracy.
Ask this paper
Construction: A hidden canonical event log drives four system views through a separately faulted delivery stream, so the disagreement follows from specified fault semantics rather than being written by hand.
Grading: An episode is scored by the merchant's terminal economic position relative to a privileged reference that knows when the pending capture resolves. Some actions cannot be undone, which is what makes the timing decision costly.
Twin pairs expose a metric failure: 321 tasks include 45 twin pairs whose four system views are identical at the decision instant and whose authoritative probes both return unknown, but whose correct dispositions differ. Over 14,445 graded episodes from nine policies, ranking by single-task accuracy and by paired loss disagree in seven places. A ship-on-first-sign policy is second-best by accuracy at 65.7% and worst in the suite by paired loss, because it cannot distinguish twins.
What works: Gating irreversible actions on an authoritative finality probe reaches 85.4% and loses nothing to pass^5, unlike every polling policy. Language models reach the same exact rate as the hand-written gate on a stratified subset, lose about twice as much money, and find the finality-gating strategy without being told it.
Abstract
A merchant's payment processor, ledger, ERP and bank feed are updated by messages that get delayed, duplicated, dropped and reordered, so for minutes at a time the four hold contradictory beliefs about the same order. An agent resolving the exception must decide whether to ship goods, re-submit a capture, refund or wait, knowing some of those cannot be undone. We present FinalityBench, an executable benchmark for that decision. It keeps a hidden canonical event log and derives each system's view from a separately faulted delivery stream, so disagreement follows from specified fault semantics rather than being authored. Grading is on executed monetary effects: an episode is scored by the merchant's terminal economic position, relative to a privileged reference told when the pending capture resolves. The corpus of 321 tasks includes 45 twin pairs (90 tasks): tasks whose four system views are identical at the decision instant, whose authoritative probes both return unknown, and whose eventual correct dispositions differ. That snapshot indistinguishability is checked under every evaluation seed rather than assumed; equivalence over all interaction traces is not claimed. Over 14,445 graded episodes from nine programmatic policies, ranking by single-task accuracy and by paired loss disagree in 7 places: a ship-on-first-sign policy is second-best by accuracy at 65.7% and worst in the suite by paired loss, because it cannot tell the two members apart. A runtime gating irreversible actions on an authoritative finality probe reaches 85.4% and, unlike every polling policy, loses nothing to pass^5; its residual loss is almost entirely one archetype, which prices finality information directly. Language models reach the same exact rate as the hand-written gate on a stratified subset, lose about twice as much money, and discover the finality-gating strategy without being told it.