Agent S3
Free while signed in. Answers cite the passages they came from.
The paper introduces Behavior Best-of-N (bBoN): run many full CUAs in parallel, convert each rollout into a compact behavior narrative, then do comparative selection to pick the best trajectory. With a stronger base agent (Agent S3), this sets the state of the art on OSWorld and generalizes to Windows and Android.
Behavior Best-of-N: sample multiple complete rollouts, summarize each with before/after deltas and pointer crops, and select the winner via a one-shot MCQ judge.
Agent S3 baseline: a flatter loop with an integrated coding sub-agent increases success and cuts LLM calls and wall time compared to Agent S2.
Results: new SoTA on OSWorld at 100 steps, with strong gains in efficiency, and the approach transfers to Windows and Android setups.
Scaling: accuracy rises as N grows, model diversity improves Pass@N, and single-round comparative selection matches or beats pairwise tournaments at lower cost.
Practical takeaways: spin up parallel VMs from the same snapshot, instrument steps to emit verifiable deltas, start with N around 4 to 10, and add diverse strong models if budget allows.
Limitation: assumes independent parallel runs; shared real-desktop side effects can leak across attempts.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack