DeskForge: Dense Supervision from Desktop Environments for Computer-Use Agents
A. Said Gurbuz, Ahmed Nassar, Sunghwan Hong, Marc Pollefeys and Peter W. J. Staar (ETH Zurich, IBM Research Zurich, Microsoft) build DeskForge, a controllable desktop environment that composes real applications and records dense annotations, and release the DeskForge-1M grounding corpus.
Ask this paper
Corpus. 1.2M annotated desktop observations with 159.7M element instances and 917K recorded click transitions across 19 applications, seven appearance presets and seven resolutions. 90.9% of screens show at least two applications and 97.7% contain occluded elements.
Grounding gains. Fine-tuning four vision-language models improves every held-out condition and raises mean accuracy on five external GUI grounding benchmarks by 1.5 to 24.9 points. Qwen's held-out accuracy goes from 76.26% to 87.54%, with +11.51 on ScreenSpot-Pro and +10.11 on OSWorld-G.
Harder scenes, larger gains. As the number of applications on screen grows from one to four or more, the fine-tuned model loses only a few points, and its margin over the base model widens from 9.7 to 14.3 points.
Downstream tasks. With a fixed Qwen3.6-27B planner and only the action model changed, tasks solved rise from 31 to 50 on a 119-task WebArena-Infinity panel and from 3 to 15 on the 100-task OpenApps set.
Abstract
Computer-use agents need to reliably ground action targets in complex desktop scenes, where multiple applications, overlapping windows, and visually similar controls compete for attention. Existing training data rarely pair such scenes with dense annotations or vary them in a controlled way. We introduce DeskForge, a controllable desktop environment that composes and explores real applications to generate large-scale supervision for computer-use agents. It varies application states, content, window layout, appearance, and resolution, and fuses screenshots, accessibility trees, and window geometry into dense element annotations while recording the outcome of each executed action. Using this environment, we construct DeskForge-1M, a corpus of 1.2M annotated desktop observations containing 159.7M element instances. We fine-tune four vision-language models on 200K grounding examples drawn from DeskForge-1M. All four improve across held-out desktop conditions and on all five external GUI grounding benchmarks; for Qwen3.5-4B, accuracy increases by 11.51 percentage points on ScreenSpot-Pro and 10.11 points on OSWorld-G. The gains also translate to long-horizon task completion: under a fixed planner, the fine-tuned action models solve more WebArena-Infinity and OpenApps tasks, with Qwen3.5-4B increasing from 31 to 50 of 119 tasks and from 3 to 15 of 100 tasks, respectively. These results show that controllable composition of real desktop environments provides a scalable source of supervision for improving both GUI grounding and long-horizon computer use. The framework code, the dataset, and the fine-tuned model are available from the project page: https://saidgurbuz.github.io/deskforge/