Coding Agents are Strong Prompt Optimizers
Agamdeep Singh, Sumit Gulwani and colleagues at Microsoft show that an off-the-shelf coding agent given a static corpus of agent trajectories can write an optimized prompt in one pass, with no environment access or validation data.
Ask this paper
CASD. Coding-Agent Skill Distillation has the coding agent write and run analysis code over the whole trajectory corpus, compute corpus-wide statistics, find systematic failure modes, inspect representative episodes and turn the findings into behavioral rules.
Reflection scope. The authors attribute the gain to analyzing the full corpus at once, where reflective optimizers look at a small batch of trajectories per step.
Results. Across ALFWorld, tau2-bench retail and telecom and SpreadsheetBench-Verified, one CASD pass beats GEPA on three of four benchmarks and SkillOpt on all four, improving the unoptimized baseline by 16.6 points on average against 10.9 for GEPA and 5.3 for SkillOpt.
Cost. One optimized prompt costs about $1.60, more than 22x cheaper than validation-gated search.
With extra advantages for baselines. When competing methods get more validation data and unrestricted environment access, CASD still leads on two of four benchmarks.
Abstract
Search-based prompt optimizers improve prompts through iterative search: they propose edits, execute fresh rollouts, score the resulting trajectories, and retain only edits that improve a validation metric. We show that this optimization loop is unnecessary. Given only a static corpus of agent trajectories, an off-the-shelf coding agent can directly synthesize an optimized prompt, requiring neither environment access nor validation data. We call this approach \textit{Coding-Agent Skill Distillation} (CASD). The key insight is reflection scope. Rather than reasoning over a small batch of trajectories at each optimization step, the coding agent writes and executes analysis code to compute corpus-wide statistics, identifies systematic failure modes, inspects representative episodes, and distills the resulting insights into behavioral rules. Across four agentic benchmarks (ALFWorld, $τ^2$-bench retail and telecom, and SpreadsheetBench-Verified), under matched data access, a single CASD pass outperforms GEPA, a state-of-the-art reflective prompt optimizer, on three of four benchmarks and outperforms validation-gated reflective search (SkillOpt) on all four, improving the unoptimized baseline by 16.6 percentage points on average versus 10.9 for GEPA and 5.3 for SkillOpt. Because CASD performs a single offline analysis pass rather than iterative search, producing an optimized prompt costs approximately \$1.60---over $22\times$ cheaper than validation-gated search. Even when competing methods are granted additional validation data and unrestricted environment access, CASD remains ahead on two of four benchmarks. These results suggest that corpus-scale statistical reflection is a viable alternative to iterative search for prompt optimization.