When Interfaces Speak: Data-Aware Generative UI Harness for Active Interaction
Xiaolong Li, Reynold Cheng and colleagues (The University of Hong Kong, with Microsoft Research and the BIRD team) present GenUI-Harness, which pairs a Tool Agent with a GUI Coder Agent that writes temporary interfaces to collect structured input from users, plus UI-TAU Bench to evaluate it.
Ask this paper
Two agents. The Tool Agent retrieves data and executes actions. The GUI Coder Agent identifies what is still ambiguous and generates front-end code for an interface that asks the user for it.
RL infrastructure. Dynamic UX runs interaction and reward collection in one sandbox, raising throughput 18.2% and cutting Chromium process count 59.4%. A Reward Auditor monitors LLM-judge reward distributions for hacking patterns and updates a shared rubric.
UI-TAU Bench. Ten database-backed domains built on tau-bench tool-use settings, with 300-task Lite and 1,000-task Full splits.
Results. Training a 4B backbone in the harness raises Pass@3 from 9.33% to 58.00%, above Claude Opus 5 at 46.67%. Across backbones the harness adds 4.48 points of Pass@3 over smolagents with 36% to 43% fewer turns.
Fewer rounds. In a reviewer study, generated UIs reduce dialogue rounds from 3.4 to 1.2.
Abstract
Most human-agent interaction today remains text-based. Natural language can impose cognitive overload, ambiguity, information chaos, and slow input for complex tasks; ephemeral generative UIs can present structured information and guide users toward task completion. We propose GenUI-Harness, a multi-agent harness pairing a Tool Agent for information retrieval and task execution with a GUI Coder Agent that identifies ambiguities and generates front-end code for structured interfaces. Training the coder with reinforcement learning is challenging: verifiable rewards for interactive UI generation require costly execution, while LLM-as-a-Judge rewards are prone to reward hacking. We address the first challenge with Dynamic UX, a lightweight package for dynamic interaction and reward collection in a single sandbox, and the second with Reward Auditor, a meta-reward mechanism that monitors reward distributions and distills diagnostic patterns into a shared rubric and scoring specification. We introduce UI-TAU Bench, a benchmark for active human-agent interaction through generated UI code, built on 10 real-world domain databases constructed from public data sources and based on Tau-Bench tool-use settings, with Lite (300 tasks) and Full (1,000 tasks) splits. GenUI-Harness achieves an average Pass@3 gain of 4.48 percentage points over smolagents on Lite. Training with GenUI-Harness improves a 4B backbone from 9.33% to 58.00% Pass@3, outperforming larger frontier models such as Claude Opus 5 (46.67%). GenUI-Harness also remains robust on ambiguous and non-ambiguous queries. In a reviewer survey comparing communication channels, generated UIs reduce average dialogue rounds from 3.4 to 1.2. These results show that data-aware generative interfaces can support effective task completion and reduce dialogue rounds in evaluated database-backed workflows.