ActionEngine
Free while signed in. Answers cite the passages they came from.
Georgia Tech and Microsoft Research introduce ActionEngine, a training-free framework that transforms GUI agents from reactive step-by-step executors into programmatic planners. The system builds a state-machine memory through offline exploration, then synthesizes executable Python programs for task completion, achieving 95% success on Reddit tasks from WebArena with on average a single LLM call. - **Crawling Agent for offline exploration:** A Crawling Agent systematically explores web interfaces and builds a state-machine representation that captures page states, transitions, and available actions. This offline exploration phase amortizes the cost of understanding an interface across all future tasks. - **Programmatic execution:** Instead of making an LLM call at every step, the Execution Agent generates complete Python programs that navigate the state machine. This reduces costs by 11.8x and end-to-end latency by 2x compared to vision-only baselines while improving reliability. - **Vision-based fallback:** When interfaces evolve or actions fail, a vision-based mechanism detects failures, repairs actions, and updates the state-machine memory. This hybrid approach combines the efficiency of programmatic execution with the adaptability of vision-language models. - **Benchmark results:** On WebArena Reddit tasks, ActionEngine achieves 95% task success compared to 66% for reactive baselines, while requiring dramatically fewer LLM calls per task. The approach demonstrates that pre-computing interface knowledge is more efficient than reasoning about each step in real time.
Abstract
Existing Graphical User Interface (GUI) agents operate through step-by-step calls to vision language models--taking a screenshot, reasoning about the next action, executing it, then repeating on the new page--resulting in high costs and latency that scale with the number of reasoning steps, and limited accuracy due to no persistent memory of previously visited pages. We propose ActionEngine, a training-free framework that transitions from reactive execution to programmatic planning through a novel two-agent architecture: a Crawling Agent that constructs an updatable state-machine memory of the GUIs through offline exploration, and an Execution Agent that leverages this memory to synthesize complete, executable Python programs for online task execution. To ensure robustness against evolving interfaces, execution failures trigger a vision-based re-grounding fallback that repairs the failed action and updates the memory. This design drastically improves both efficiency and accuracy: on Reddit tasks from the WebArena benchmark, our agent achieves 95% task success with on average a single LLM call, compared to 66% for the strongest vision-only baseline, while reducing cost by 11.8x and end-to-end latency by 2x. Together, these components yield scalable and reliable GUI interaction by combining global programmatic planning, crawler-validated action templates, and node-level execution with localized validation and repair.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack