JoMoHa: Joint Model--Harness Optimization with Reinforcement Learning
Xudong Wu, Guangchen Lan (Salesforce AI Research) and colleagues at the University of Hong Kong introduce JoMoHa, which trains a harness-writing policy and a task-solving policy together with reinforcement learning from the same task outcomes.
Ask this paper
Problem. Naively alternating model updates and harness updates makes the two drift apart. Performance first rises and then falls as the harness and the model become incompatible.
Method. One policy writes an executable harness and another acts under it. Both are trained from the same task result, with separate token histories and value estimates, so the harness writer is learned inside the system instead of being a fixed external frontier model.
Results. JoMoHa beats weight-only and harness-only baselines and has the highest success among evaluated methods on ALFWorld, WebShop and DBBench, averaging 70.3% and reaching 97.0% on ALFWorld.
Ablations. At fixed solver weights the learned harnesses improve on the initial code, training the writer beats freezing it, and coordinating the two learners beats plain joint updates.
Abstract
The performance of language agents depends jointly on model weights and the harness. Optimizing either alone can be constrained by the other component, limiting overall performance gains. Our analysis of naive alternating training reveals progressive incompatibility between the model and harness, which increasingly interferes with task execution and may also undermine model training, leading to performance degradation. To coordinate the evolution of models and harnesses, we introduce JoMoHa, to our knowledge the first framework to jointly train an executable harness generation policy and a task execution policy through reinforcement learning. JoMoHa outperforms the evaluated weight-only and harness-only baselines and achieves the highest success rates among the evaluated methods on ALFWorld, WebShop, and DBBench, averaging 70.3% and reaching 97.0% on ALFWorld. By learning both task execution and runtime improvement from its own generated trajectories, JoMoHa performs recursive self-improvement (RSI).