Does Your Generalist Skill Optimizer Really Incentivize Agentic Tasks? Rethinking Skill Optimization via Bipartite Attribution
Jiawei Zhou, Bo Peng, Chaochao Lu and colleagues (Shanghai AI Laboratory, Fudan, SJTU, USTC, Tsinghua) introduce BiPA to audit skill optimizers for agents and find that the learned skills perform about as well as the hand-written seed while reading more tokens.
Ask this paper
Method. BiPA replays an optimizer's acceptance rule on candidates identical to the seed, which measures how much gain selection alone reports. It also records each episode as a bipartite graph of turns and the text each turn read, so extra input tokens can be traced to the behavior that caused them.
Main result. Across three optimizers, three agentic benchmarks and two models, learned skills succeed 0.1 points more than the seed on average. Keeping the best of several candidates on 30 or fewer dev tasks reports gains of this size even when no candidate changed.
Hidden cost. Learned skills read about 16% more input tokens. Optimizers almost always add rules, half of them to re-check work and some to act one call at a time, and each followed rule becomes an extra turn that re-sends the full conversation.
Four principles. Accept an edit only if its gain exceeds what selection alone produces. Charge edits for the turns they add, not their length. Let the editor delete and merge rules. Show the editor what earlier rules cost.
Abstract
Skill optimizers adapt an agent to long-horizon tasks, such as repairing a terminal environment, by rewriting its skill, a natural-language procedure in its context, and keeping the edits that raise success on a small dev set. On such tasks success is noisy, so a gain on a small dev set does not show that an edit helps, and a success score does not show what the edit costs. We introduce BiPA, which replays an optimizer's acceptance rule on candidates identical to its seed to measure what selection alone reports, and records each episode as a bipartite graph of turns and the text they read to trace extra input tokens to the behavior behind them. Across three optimizers, three agentic benchmarks and two models, the learned skills succeed about as often as their hand-written seed, 0.1 points higher on average. Their dev gains vanish on test: keeping the best of several candidates on at most 30 tasks reports gains this large even when none changed. They read about 16% more input tokens: the optimizers almost always add rules, half to re-check work and some to act one call at a time, and the agent follows them as extra turns, each re-sending the whole conversation. Four principles follow: accept an edit only if its gain exceeds what selection alone reports; charge it for the turns it adds, not for its length; let the editor delete and merge rules; show it what earlier rules cost. Code is available at https://github.com/valleysprings/BiPA.git.