CoAct-1
Researchers from USC, Salesforce, and UW present CoAct-1, a multi-agent system that combines GUI interaction with direct code execution to improve efficiency and robustness in computer-using agents. Unlike prior GUI-only frameworks, CoAct-1’s Orchestrator delegates subtasks to either a GUI Operator (vision-language action model) or a Programmer (Python/Bash execution), enabling agents to bypass brittle, multi-click sequences for tasks better handled via scripts.
Ask this paper
Hybrid multi-agent architecture – The Orchestrator dynamically assigns subtasks; the Programmer writes/executes scripts for backend operations; the GUI Operator handles visual, interactive tasks. This dual-modality cuts down steps and reduces visual grounding errors.
State-of-the-art OSWorld results – Achieves 60.76% success rate (100+ step budget), outperforming GTA-1 (53.10%) and Agent S2.5 (56.00%). Excels in OS-level (75%), multi-app (47.88%), Thunderbird email (66.67%), and VLC tasks (66.07%), where code execution offers big gains.
Efficiency boost – Solves tasks in 10.15 steps on average vs. GTA-1’s 15.22, with coding actions replacing long GUI sequences (e.g., file management, data processing). Coding is most beneficial in LibreOffice Calc, multi-app workflows, and OS operations.
Backbone sensitivity – Best performance when using OpenAI CUA 4o for GUI, o3 for Orchestrator, and o4-mini for Programmer, showing gains from a strong vision model for GUI and a capable coding agent.
Limitations – Struggles with high-level queries requiring conceptual inference beyond explicit instructions and with ambiguous tasks lacking critical detail.