Harness-of-Harness: Multi-Day Autonomous Software Development with Continual Improvement
Haoyang Yan and colleagues at Shanghai AI Lab and SJTU wrap existing coding-agent harnesses in an outer loop, Harness-of-Harness, that turns multi-day autonomous software development into repeated planning, coding, and testing increments with versioned history.
Ask this paper
It sits above your harness, not instead of it. HoH operates on existing coding-agent harnesses and organizes their executions, which is why it can be demonstrated across three different harness-model pairs without reimplementation.
Five design choices carry the loop. Balance repair against capability growth, scope work into small verifiable increments, separate implementation-time testing from independent evaluation, constrain verifiable outputs rather than prescribing workflows, and progressively expose deliverables, role-specific tools, and skills.
Reuse over recreation. Versioned project histories and progressive exposure are what stop each iteration from rebuilding what the last one produced, which is the standard failure of long-running autonomous development.
Large consistent gains. Across GameCraft-Bench, FrontierSWE, and ProgramBench with Codex plus GPT-5.5, OpenCode plus DeepSeek-V4-Pro, and Pi plus MiniMax-M3, HoH averages a 52.25% relative gain over the standalone harnesses after three iterations, peaking at 82.86%.
The demo is the argument. A multi-day deployment of more than 70 iterations autonomously produced a playable first-person shooter with a coherent storyline, implemented core mechanics, polished visuals, and integrated audio.
Abstract
This paper studies autonomous software development, in which LLM-based coding agents transform high-level requirements into complete, functional, and usable software systems without human intervention. We introduce Harness-of-Harness (HoH), a framework that enables coding agents to continually improve software during autonomous development. HoH operates on existing coding-agent harnesses, and organizes their executions into iterative planning-coding-testing loops. To sustain improvement across loops, HoH balances repair with capability growth, scopes development into small and verifiable increments, separates implementation-time testing from independent evaluation, and constrains verifiable outputs rather than prescribing agent workflows. It progressively exposes deliverables, role-specific tools, and skills, encourages reuse rather than recreation, and maintains versioned project histories. On GameCraft-Bench, FrontierSWE, and ProgramBench, three harness-model pairs (Codex with GPT-5.5, OpenCode with DeepSeek-V4-Pro, and Pi with MiniMax-M3), HoH consistently outperforms the corresponding standalone harnesses, achieving an average relative gain of 52.25 percent and a maximum gain of 82.86 percent after three iterations. In a multi-day deployment with more than 70 iterations, HoH autonomously develops a first-person-shooter game, featuring a coherent storyline, fully implemented core mechanics, human-playable experience, polished visuals and integrated audio. Github: https://github.com/Flesymeb/HarnessOfHarness Project Page: https://flesymeb.github.io/HarnessOfHarness/