ASIL: Replacing Screenshot-and-Click with Structured State and Semantic Actions
Rui Xie and Lu Chen (EMNLP Findings) argue that screenshot-and-click is the wrong interface for software-operating agents and build ASIL, which exposes applications through structured JSON observations and code-executable semantic actions.
Ask this paper
The indictment of pixels: Screenshots are state-incomplete and GUI actions are brittle, semantically weak, and poorly matched to long-horizon planning. ASIL instead uses the deepest feasible access path per application.
The gap is enormous: Across 15 applications and 300 single-app plus 80 multi-app tasks, ASIL exceeds 80 with closed models in fewer than five actions per task. The same tasks under screenshot-and-click with a 50-step budget yield 6.6 and 26.6 strict success.
Beats some native APIs too: ASIL exceeds LibreOffice's UNO API by 28 to 38 strict points on matched tasks, though it only matches draw.io's MCP content contract.
The structured modality trains well: Small-scale SFT lifts Qwen3.5-2B from 58.0 to 72.1 and Qwen3.5-9B from 66.6 to 80.4; resource-limited on-policy RL takes them to 74.4 and 82.2.
Why it matters: The pixel-versus-protocol argument for computer use is being settled empirically, and this is one of the sharper data points for building agent-native access layers instead of teaching agents to squint at screenshots.
Abstract
Powerful code agents can execute scripts, call tools, and manage files, yet many important applications remain accessible primarily through graphical user interfaces. We argue that screenshot-and-click is an inefficient interface for software-operating agents: screenshots are state-incomplete, and GUI actions are brittle, semantically weak, and poorly matched to long-horizon planning. We introduce ASIL (Agent-Software Interaction Layer), an agent-native interface that exposes software through structured JSON observations and code-executable semantic actions, realized through the deepest feasible access path for each application. We instantiate ASIL across 15 applications and a benchmark of 300 single-application and 80 multi-application tasks. ASIL reaches above 80 with closed models while executing fewer than five actions per task. Under a repaired runtime and a 50-step screenshot budget, the same tasks yield 6.6 and 26.6 strict success under screenshot-and-click control, rising to 15.0 and 53.3 on an easier OSWorld-comparable band. Against application-native interfaces on matched tasks, ASIL exceeds LibreOffice's UNO API by 28-38 strict points but only matches draw.io's MCP content contract. The structured modality also suits training: small-scale SFT raises Qwen3.5-2B from 58.0 to 72.1 and Qwen3.5-9B from 66.6 to 80.4, and resource-limited on-policy RL further raises them to 74.4 and 82.2.