Code Understanding is a Bottleneck for Coding Agents
Nishant Balepur, Kiran Tomlinson and Tobias Schnabel (Microsoft Research, with the University of Maryland) present CABRA, a synthetic benchmark that builds coding tasks as call-graph transformations to isolate which abilities drive coding-agent errors.
Ask this paper
Design. CABRA scales difficulty with a task-size parameter along four axes: function traversal, search, runtime resolution and instruction following.
Scale. Eight LLMs and six coding agents are run on 6,840 CABRA tasks.
Tools hide weakness. Bare LLM accuracy falls as task size grows, but agents stay near-perfect because they offload the work to tools such as grep.
Difficulty signal. Larger tasks elicit more reading and analysis tool calls, and on SWE-bench Verified those tool-call counts predict agent accuracy better than lines of code edited, so difficulty lies in understanding the code to change.
Hard case. An extension where models must analyze divergent logic across two classes finally lowers agent accuracy, which the authors use to argue for pairing SWE-bench-style tasks with controlled synthetic diagnosis.
Abstract
Repository benchmarks (e.g., SWE-bench) for coding agents often assume that lines of code edited can predict task difficulty, but such datasets' poor control over code and task types makes it hard to know which abilities truly drive agent errors. We present CABRA: a Coding Ability Blueprint for Rigorous Agent evaluation. CABRA builds tasks from scratch as call graph transformations and scales difficulty via a task size parameter on four axes: function traversal, search, runtime resolution, and instruction following. We run eight LLMs and six coding agents on 6,840 CABRA tasks to show: 1) LLM accuracy falls as task size~grows, but agents stay near-perfect by offloading work to tools (e.g., grep); 2) Larger CABRA tasks elicit more tool calls for reading and analysis, while a separate study on SWE-bench Verified shows these tool call counts predict agents' accuracy better than lines of code edited, suggesting task difficulty for agents can lie in understanding code to edit, not just in making edits; 3) Extending CABRA to an intense understanding task where models analyze divergent logic across two classes backs this finding, as agent accuracy finally falls. More broadly, we argue for synthetic evaluations like CABRA to unmask LLM weaknesses trivialized by tools (e.g., needle-in-a-haystack) and abilities beyond just editing (e.g., understanding) that coding agents still find difficult, pairing SWE-bench-style tasks with controlled diagnosis.