How Many Pixels Is a Digit Worth? Place-Aware Coordinate Entropy for GUI Agent Confidence Estimation
Yunxiang Li, Xixin Wu and Helen Meng (The Chinese University of Hong Kong, EMNLP 2026 Main) show that GUI agents' click confidence improves when each coordinate digit's entropy is weighted by its place value.
Ask this paper
Problem. GUI agents emit click coordinates as sequences of digit tokens, and standard text-LLM confidence methods separate correct clicks from wrong ones only weakly. GUI-specific methods need K samples or extra supervision.
Place-value asymmetry. Whether a click lands inside a bounding box depends mostly on the higher-place digits, so averaging uncertainty uniformly across digits dilutes the relevant signal.
PACE. Place-Aware Coordinate Entropy weights each digit's Shannon entropy by its place value and needs a single forward pass.
Results. On ScreenSpot-Pro and ScreenSpot-v2 with fixed-scale agents, PACE wins both AUROC and selective accuracy in all primary comparisons and matches or beats K-sample baselines at a fraction of their cost.
Abstract
GUI agents predict click coordinates as digit-token sequences, but standard text-LLM confidence estimation methods rank correct clicks from wrong ones only weakly. GUI-specific alternatives use K samples or new supervision, but still leave room for improvement. We trace part of this to place-value asymmetry: bounding-box correctness often makes higher-place digits more important than lower-place digits, so uniform aggregation weakens the signal that determines correctness. The fix is to weight each digit's Shannon entropy by its place value. We call this Place-Aware Coordinate Entropy (PACE). Across fixed-scale agents on ScreenSpot-Pro and ScreenSpot-v2, PACE wins both AUROC and selective accuracy on all primary comparisons in a single forward pass, matching or outperforming K-sample baselines at a fraction of the cost. PACE provides a per-click confidence estimate that turns coordinate-token internals into a practical confidence signal for GUI agent deployment.