CUA-Universe: A Scalable and Dynamic Environment for Hybrid GUI+CLI Agents
Haoting Shi and colleagues at Shanghai Jiao Tong build a pipeline that converts real desktop software into environments where an agent can act through both the GUI and the command line over shared application state.
Ask this paper
The observation. Real computer work is hybrid: visual state inspection plus precise high-throughput commands. CLI-native agents lack visual perception for layout-dependent tasks and GUI-native agents are inefficient at operations a command handles directly.
App-Forge. Adapts applications into reproducible VMs and command-line surfaces it discovers, wraps, or generates, scaling to 16 applications without per-app manual engineering.
Task-Weave and Path-Steer. Task-Weave synthesizes hybrid tasks of controllable difficulty from reusable operations over seed files; Path-Steer steers rollouts along efficient hybrid paths and harvests verified trajectories for post-training.
Results. A 9B model trained on the data improves on CUA-Verse by 39.3 points with 37% fewer steps and 60% fewer tokens, on OSWorld by 16.8 points with 57% fewer steps and 44% fewer tokens, and on OSWorld-MCP by 7.84 points.
The behavioral shift. Training moves the agent away from inefficient GUI clicking and brittle CLI scripting toward choosing the cheaper modality per step.
Abstract
Computer-use agents have advanced on benchmarks like OSWorld and AndroidWorld, but still act mostly through the GUI, often producing inefficient trajectories. Real-world computer work is hybrid, combining visual-state inspection with precise, high-throughput command-line operations, so capable agents must coordinate both modalities over shared application state. Yet scalable hybrid environments remain scarce because supporting both GUI and CLI over real applications typically requires substantial manual engineering for each application. Existing agents also struggle to use the two interfaces complementarily: CLI-native agents lack visual perception for tasks involving interface state or layout, while GUI-native agents are inefficient for operations better executed through commands. We introduce CUA-Universe, a scalable environment-to-data pipeline that turns real desktop software into hybrid GUI+CLI environments. App-Forge adapts applications into reproducible VMs and command-line surfaces it discovers, wraps, or generates, scaling to 16 applications; Task-Weave synthesizes diverse hybrid tasks of controllable difficulty from reusable operations over seed files; and Path-Steer steers rollouts along efficient hybrid paths and harvests verified trajectories for post-training. Training on this data shifts behavior from inefficient GUI interaction and brittle CLI scripting toward effective GUI+CLI orchestration. Our 9B model improves both success and efficiency on CUA-Verse (Score +39.3 pts; -37% steps, -60% tokens), OSWorld (SR +16.8 pts; -57% steps, -44% tokens), and OSWorld-MCP (Score +7.84 pts; -27% steps, -30% tokens). CUA-Universe provides a scalable path toward more capable and efficient computer-use agents.