Selective Critique for Cost-Aware LLM Agents in Long-Horizon Decision Making
Heewon Park, Somin Im and Minhae Kwon (Soongsil University) propose SAG, a training-free gate that decides at each step whether an LLM agent should call a critic, using action-level ambiguity as a proxy for the value of the critique. Accepted to NeurIPS 2026.
Ask this paper
Gating signal. SAG scores ambiguity over admissible actions with global entropy and the local top-2 margin, and calls the critic only when the estimated value of information exceeds its cost.
ALFWorld result. Task success rises from 24.6% to 78.4% at a token budget comparable to ReAct, a 3.1x improvement in normalized token efficiency over always-on critique.
Small models. A 7B actor with a 3B critic matches a 14B actor without critique.
Online self-improvement. The actor is fine-tuned online on critic-assisted behavior, so it needs the critic less over time.
Scope. Results cover ALFWorld, BabyAI-Text and WebShop with several backbones; the appendix documents failure modes such as wrong critic advice and early errors the critic cannot repair.
Abstract
Improving the reliability of large language model (LLM) agents in long-horizon decision-making remains a key challenge. When deployed as autonomous agents interacting with complex environments, early mistakes can propagate through trajectories and cause cascading failures. Recent approaches improve reliability by incorporating external critique or deliberation, but invoking these mechanisms at every step substantially increases token consumption and latency, limiting practical deployment. We propose SAG (Self-improving Agent with Gated critique), a cost-aware framework that formulates critique invocation as a step-wise decision problem during long-horizon interaction. SAG introduces a lightweight, training-free gating mechanism that estimates the utility of critique using action-level ambiguity signals--global entropy and local top-2 margin--computed over admissible actions. From a decision-theoretic perspective, this mechanism approximates the Value of Information (VoI) of critique, enabling the agent to selectively allocate expensive feedback only when its expected benefit justifies the cost. SAG further incorporates online bootstrapped self-improvement, allowing the actor to internalize critic-assisted behaviors and progressively reduce reliance on critique. Across three long-horizon interactive benchmarks and multiple backbone models, SAG substantially improves the performance-cost trade-off compared with both no-critique and always-on critique agents. On ALFWorld, SAG increases task success from 24.6% to 78.4% while maintaining a token budget comparable to ReAct, yielding a $3.1\times$ improvement in normalized token efficiency. Moreover, a 7B actor with a lightweight 3B critic achieves performance comparable to a 14B actor without critique, showing that selective critique can recover most of the reliability benefits of deliberation while dramatically reducing inference cost.