AutoHarness
Free while signed in. Answers cite the passages they came from.
Google DeepMind researchers introduce AutoHarness, a method for automatically synthesizing code harnesses that prevent LLM agents from making illegal actions. The core insight comes from a striking observation: in the Kaggle GameArena chess competition, 78% of Gemini-2.5-Flash losses were attributed to illegal moves, not poor strategy.
Automatic harness synthesis: Rather than building complex rule systems by hand, AutoHarness lets Gemini-2.5-Flash automatically generate a code harness through a small number of iterative refinement rounds using feedback from the game environment. The harness acts as a programmatic constraint layer between the agent and the environment.
Smaller models beat larger ones: The resulting harness enables the smaller Gemini-2.5-Flash to outperform much larger models including Gemini-2.5-Pro and GPT-5.2-High on 16 TextArena single-player games. This shows that structured code constraints can compensate for raw model capability.
Complete illegal move prevention: The synthesized harness successfully prevents all illegal moves across 145 different TextArena games, covering both single-player and two-player settings. This transforms a model that previously failed on most turns into a competitive agent.
Cost-effective scaling: Using a smaller model to synthesize a custom code harness is not only more performant but also more cost-effective than simply deploying a larger model. This reframes the agent improvement problem from model scaling to harness engineering.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack