BOXBOX: Recursive Harness Self-Improvement for Long-Running Performance Optimization
Yuhao Tan and Hanyun Jiang (Nanjing University, Zhejiang University; work done at Microsoft), with Microsoft corresponding author Fangkai Yang and Nanjing's Xiaoxing Ma, introduce BOXBOX, which lets an agent revise its own harness during a long-running task, and PerfRace, a benchmark for long-running software performance optimization.
Ask this paper
Problem. Offline harness optimization tunes a harness on tasks with reference answers and reuses it. Open-ended tasks have no reference answers, and long runs expose problems that an offline-tuned harness was not built for.
Benchmark. PerfRace asks agents to speed up real software projects over hours of work, scored by measured speedup.
Method. BOXBOX lets the agent trigger a harness revision in the middle of a task. A kernel hot-reloads the revised harness while keeping the execution context, so the run continues without restarting.
Results. Under matched test-time budgets BOXBOX reaches a Speedup@4h of 1.70x, against 1.54x for the strongest baseline among a frozen harness, offline harness evolution and harness scaling, and uses task time and model cost more efficiently.
Abstract
Automated harness optimization improves agents by learning from trajectories and task feedback. Existing offline methods use reference answers to optimize harnesses for reuse on held-out tasks. However, many real-world tasks are open-ended, while long runs expose new difficulties that offline-optimized harnesses may not address. To study harness optimization in these settings, we introduce PerfRace, a benchmark for long-running performance optimization of software projects, and investigate whether agents can use task-specific experience to recursively improve their harnesses during an ongoing task. To this end, we introduce BOXBOX, which enables agent-triggered harness revision and kernel-managed hot reloading while preserving execution context. We evaluate BOXBOX on PerfRace under matched test-time budgets, comparing it with Frozen Harness, offline Harness Evolution, and Harness Scaling. BOXBOX achieves higher speedup with more efficient use of task time and model cost, reaching a Speedup@4h of 1.70x, compared with 1.54x for the strongest baseline.