Harness Learning Enables Generalizable Test-Time Adaptation
Alvin Zhang, Xuecheng Liu, Zixuan Wang, Ruslan Salakhutdinov, Daniel Khashabi, Yuda Song, Andrea Zanette and colleagues at Carnegie Mellon University train a proposer model with RL to edit an agent's executable harness from execution feedback, and show the learned revision skill transfers to tasks it never saw.
Ask this paper
Framing. Harness learning is meta-learning over programs. The proposer reads the task, the current harness code and an execution report, and writes a code edit; harness revisions play the role that weight updates play in gradient-based adaptation.
Training. The proposer is optionally initialized with SFT on teacher revisions, then trained with RL where the reward is the task score of the revised harness. The solver model stays frozen, and at test time no parameters change.
Results. On Reasoning Gym, training improves revision quality on task families excluded from both SFT and RL, and in single-step revision the trained 4B proposer beats its 35B teacher on average. A proposer trained on HotpotQA transfers to MuSiQue and 2WikiMultihopQA.
Reliability. Most of the gain on unseen tasks comes from RL, which cuts the share of proposals that produce a broken or zero-scoring harness; SFT alone leaves that rate nearly unchanged.
What it learns. On reasoning tasks the dominant learned structure is an interpreter loop that delegates computation to code; in QA, RL keeps multi-hop retrieval and passes retrieved passages directly to the answer call.
Abstract
A language-model agent is jointly defined by its model and its harness, the executable program that organizes model calls, tool use, and information flow. Because different tasks call for different ways of organizing these operations, the harness needs to be adapted using feedback from the task at hand. We introduce harness learning, which trains a proposer model to revise a solver's harness using execution feedback. We formulate this process as meta-learning over executable programs, with harness revisions playing the role of weight updates in gradient-based adaptation. We train the proposer with reinforcement learning, using the task performance of revised harnesses as the reward. At test time, the proposer uses feedback from successive executions on a new task to refine the harness, without performing any parameter-space update. Experiments on reasoning and multi-hop question answering show that harness learning improves revision quality and that the ability to adapt at test time transfers to unseen tasks. Policies trained on individual revisions can continue improving harnesses over multiple rounds, while the benefits of training on revision sequences vary across settings. These findings suggest a path towards continually learning agents that turn accumulated experience into generalizable improvements.