ParetoSaddler: Pareto-Guided Optimization of Agent Harnesses for Quality and Cost
Xingzhi Qian, Jue Zhang and colleagues at Microsoft (with intern authors from UCL, POSTECH and other universities) present ParetoSaddler, a harness optimizer that improves agent quality and cost together instead of quality alone.
Ask this paper
Problem. Applying the deployment preference (for example, minimum cost at a quality floor) at every optimization step rejects intermediate harnesses that would have led to better ones later.
Method. An optimizer edits a parent harness into a candidate, runs both repeatedly, and promotes the candidate if a Bayesian bootstrap over the runs gives credible Pareto progress in either quality or cost. Promoted candidates get a broader development evaluation and stay available as parents. Deployment selection happens only at the end.
Results. On ClawMark, quality rises from 70.86% to 81.35% while cost per task falls from $1.17 to $1.01. On GAIA2, quality rises from 56.22% to 62.89% while cost falls from $0.78 to $0.45. Splits are domain-disjoint.
Comparison. The Pareto-selected harnesses reach higher quality at lower cost than harnesses from quality-only optimization.
Ablations. Pareto-aware promotion keeps candidates that a deployment-preference gate would discard and that later produce improvements. Bayesian screening removes candidates whose gains do not hold across repeated runs.
Abstract
Harness optimization can substantially improve LLM agents without modifying model weights, but existing methods primarily optimize response quality. Extending them to joint quality--cost optimization introduces a challenge: applying the deployment preference during intermediate updates can prematurely reject candidates that are valuable for later improvements. We introduce ParetoSaddler, which addresses this mismatch by separating intermediate candidate promotion from final deployment selection. At each iteration, an optimizer updates a parent harness to produce a candidate, and then compares the two over repeated executions and uses Bayesian bootstrap Pareto support to promote candidates showing credible progress in either quality or cost. Promoted candidates receive broader development evaluation and remain available as parents for later optimization steps. We evaluate ParetoSaddler on ClawMark and GAIA2 using domain-disjoint splits. On ClawMark, ParetoSaddler improves quality from 70.86\% to 81.35\% while reducing cost from \$1.17 to \$1.01 per task; on GAIA2, it improves quality from 56.22\% to 62.89\% while reducing cost from \$0.78 to \$0.45 per task. The Pareto-selected harnesses also achieve higher quality at lower cost than quality-only harness optimization. Analysis shows that Pareto-aware optimization retains intermediate candidates that would be discarded by deployment preference but can support later improvements, while Bayesian bootstrap screening filters out candidates whose gains are weakly supported across runs.