Closing the Consistency Gap: Self-Evolving Agents That Learn to Stay on Course
Evelyn Duesterwald, Benjamin Elder, Lilian Ngweta, Shashanka Ubaru and Malgorzata Zimon at IBM Research name the consistency gap, the difference between an agent's average pass rate and how often it succeeds on all five repeats of the same task, and close part of it with targeted episodic memory.
Ask this paper
The gap, measured: A ReAct agent on AppWorld using GPT-4.1 averages a 77% per-run pass rate but succeeds in all five runs only 53% of the time, a 24-point shortfall that average accuracy hides completely.
Why it is the deployment-relevant number: An agent that is right most of the time but not repeatably is not trustworthy in production, so the authors treat closing this gap as a precondition rather than an optimization.
A Consistency Analyzer locates the instability: It pinpoints where and why a trajectory is likely to flip across executions, which is a different question from where the trajectory failed once.
A Guideline Generator converts diagnosis to memory: The diagnosis becomes targeted guidelines committed to episodic memory and injected into future executions on similar tasks.
Result: On AppWorld with ReAct and GPT-4.1 the fraction of tasks succeeding in all five runs rises by 16 points on same-task evaluation and 13 points on similar-task generalization.
Abstract
Large language model (LLM)-powered agents can be accurate on average yet unreliable in production, a discrepancy that has been observed but remains largely unaddressed. When given the same task five times, a ReAct agent on the AppWorld benchmark using GPT-4.1 succeeds in all five runs only 53% of the time, even though its per-run pass rate averages 77%. We call this 24-point shortfall the consistency gap, and we argue that addressing it is a precondition for trustworthy AI agent deployment. We present a self-evolving agent framework that reduces this gap by identifying unstable, low-consistency steps in agent trajectories and converting them into episodic memory the agent can draw on in future runs. At its core is a Consistency Analyzer that pinpoints where and why a trajectory is likely to flip across executions, and a Guideline Generator that converts the diagnosis into targeted guidelines, committed to memory and injected into future agent executions on similar tasks. On AppWorld with ReAct/GPT-4.1, our framework raises the fraction of tasks that succeed in all five runs by +16 points on same-task evaluation and +13 points on similar-task generalization.