CURA: Certified Runtime Alarms for Computer-Use Agents
Divake Kumar and colleagues (UIC with Intel Labs) show that computer-use agents claim success on 90% of their own failures, then build an external monitor that reads only harness-visible telemetry and turns a running trajectory into a sequential test with certified false-alarm control.
Ask this paper
The self-report finding: On 361 OSWorld tasks their pipeline scores 82.9 mean, above the 72.4 human reference, yet 64 of 71 failures end with a success claim, 61 acknowledging no blocker, and the explicit failure affordance is never used in roughly 9,100 calls.
No model internals: CURA needs no activations, no extra LLM calls, and no prompt changes, which makes it deployable against a closed agent.
Early warning: At alpha 0.10 the CUSUM alarm catches 42.3% of failures a median of 31 steps before termination at a realized false-alarm rate of 0.066, and a read-only gate probe resolves some risk before the first action (0.69 AUROC).
Honest negative: Retrospectively the composite reaches 0.828 AUROC but its margin over a plain total-token baseline is not significant (+0.026, p = 0.101); the separation is online, at matched certified budgets.
Deployable cascade: Alarm-gated mid-execution oversight recovers 23 of 70 failures while spending a frontier overseer on only 38, lifting the pipeline to 86.8 mean and 84.5% full-solve.
Abstract
Self-report is the cheapest oversight channel a deployer has, and on capable computer-use agents (CUAs) it fails precisely where oversight matters. On 361 OSWorld tasks our pipeline, a read-only feasibility gate, a planner, and a GUI executor, reaches a mean task score of 82.9, above the 72.4 human reference, yet 64 of its 71 failures (90%) end with a success claim, 61 acknowledging no blocker, and the explicit failure affordance is never used in roughly 9,100 calls. We introduce CURA (Certified Runtime Alarms for Computer-Use Agents), an external monitor that reads only harness-visible telemetry, with no model internals, extra LLM calls, or prompt changes, and turns the running trajectory into a sequential test with certified false-alarm control. At alpha = 0.10 its CUSUM alarm detects 42.3% of failures a median of 31 steps before termination at a realized false-alarm rate of 0.066, and risk is partly resolvable before the first action (gate probe, 0.69 AUROC). Retrospectively the composite reaches 0.828 AUROC (fold-internal floor 0.802), but its margin over a total-token baseline is not significant (Delta = +0.026, p = 0.101); the separation is online, where CURA recalls more at matched certified budgets: 0.41 versus 0.34 at alpha = 0.10, 0.56 versus 0.38 at alpha = 0.20. Alarm-gated mid-execution oversight recovers 23 of 70 failures while spending a frontier overseer on 38, giving a deployable cascade at mean score 86.8 and 84.5% full-solve (305 of 361). The certificate bounds false alarms only. We also report where behavioral monitoring is uninformative.