Online Dense Reinforcement Learning with Planning-Grounding Agent for UI Navigation on Live IOS Apps
Peitong Duan, Khashayar Gatmiry, Jeffrey Nichols, Samy Bengio and Emmanuel Abbe (Apple) train UI navigation agents with online reinforcement learning on live iOS apps, using a learned planner, a frozen grounding model and an LLM judge that scores each step.
Ask this paper
Split agent. An RL-trained Planning agent is paired with a frozen SFT grounding model, which focuses open-ended reasoning on the planner.
Dense reward. An LLM judge supplies step-level and terminal rewards, trained with PPO and asymmetric discounting to prevent reward hacking. GiGPPO uses visually similar states across concurrent trajectories to stabilize the critic.
Real iOS apps. On UINavBench, the 8B PPO agent ties Gemini-3.1-Pro at 32.4% and beats Gemini-3.5-Flash at 28.4% and Qwen3.8-27B at 19.6%.
Transfer to Android. Trained only on iOS, it reaches 32.8% on AndroidWorld against 4.7% for the SFT baseline, nearly a sevenfold gain.
Remaining gap. Gemini-3.1-Pro still leads on AndroidWorld at 60.8% and on iOSWorld at 39.5%, where the best RL checkpoint reaches 9.8%.
Abstract
While frontier vision-language models demonstrate strong capabilities in user interface (UI) navigation, their high inference costs prohibit everyday use. Conversely, smaller models trained via supervised fine-tuning (SFT) or offline reinforcement learning often memorize static trajectories, leading to brittle execution and an inability to recover from errors. To address these limitations, we propose a novel online reinforcement learning framework that trains UI navigation agents directly on live iOS applications. Our framework decouples the agent into two specialized components: a Planning agent and a UI Grounding model. We rely on an LLM-as-a-Judge to provide comprehensive reward signals, combining dense, step-level feedback to evaluate intermediate progress with a terminal reward for overall task completion. To effectively leverage this dual reward structure, we optimize the agent using Proximal Policy Optimization (PPO) with step-level advantage estimation. Furthermore, we introduce GiGPPO, a novel neighbor-conditioned value estimation method that improves critic stability by leveraging reward signals from visually similar states across concurrent trajectories. To support this online training paradigm at scale, we introduce an efficient infrastructure that orchestrates hundreds of concurrent virtual iOS devices. Evaluations on rigorous benchmarks, including UINavBench, iOSWorld, and AndroidWorld, demonstrate that our online RL trained agents substantially outperform both SFT and terminal-only RL baselines while accelerating training convergence. Notably, our 8B RL agents achieve nearly a sevenfold improvement over the SFT baseline in out-of-distribution generalization to Android environments, while outperforming Gemini-3.5-Flash and tying with Gemini-3.1-Pro on real iOS applications.