MoMHa: Multi-Objective Optimization of LLM Harnesses over Accuracy, Safety, and Tokens
Subhojyoti Mukherjee and Md Mehrab Tanjim at Adobe Research (NeurIPS 2026) treat the harness, the code that builds prompts, routes calls and parses outputs, as a design target optimized jointly for accuracy, behavioral safety and token cost.
Ask this paper
Search setup. An agentic proposer (Claude Code) with filesystem access to prior harness source, execution traces and scores proposes new harnesses. The headline scoring treats a 1k-token reduction as worth 1 point of accuracy.
Joint beats staged. A single-phase proposer that sees all three objectives at once (MoMHa) outperforms a two-phase accuracy-then-tokens variant, scalar-only feedback and an accuracy-only baseline.
Synthetic track. Joint mean of 0.482 against 0.198 to 0.422 for ten baselines, winning 7 of 10 per-domain columns.
Real-world transfer. On seven public benchmarks (HumanEval, MBPP, Spider, FEVER, MMLU-Pro, LawBench, NuminaMath) it scores 0.461 against 0.377 for DSPy, winning 5 of 7.
Scope. Seventeen domains in total, including three user-specific safety domains derived from U-SafeBench, and a 12-model fleet across four families.
Abstract
Most work on improving large language models treats accuracy as the sole objective. We argue that the harness, the Python code surrounding the model that constructs prompts, routes calls, and parses outputs, is a first-class design surface whose quality is inherently multi-objective: an accurate harness that refuses no unsafe request, or that consumes an order of magnitude more tokens, is not a good harness. We present Meta-Harness, a system that casts harness design as search over three per-domain objectives (accuracy, behavioural safety, and token cost) solved by an agentic proposer (Claude Code) with full filesystem access to prior harness source, execution traces, and scoring artifacts. Our central finding is that a singlephase joint-reward proposer (MoMHa) outperforms every alternative, including a two-phase "accuracy then tokens" ablation, scalar-only feedback, and an accuracy-only baseline. We evaluate on seventeen domains: seven synthetic capability suites, seven real-world public benchmarks (HumanEval, MBPP, Spider, FEVER, MMLU-Pro, LawBench, NuminaMath), and three U-SafeBench-derived user-specific safety domains, using a 12-model fleet spanning four families. On the synthetic track MoMHa achieves a joint mean of 0.482 versus 0.198-0.422 for ten baselines, winning $7 / 10$ per-domain columns; on the real-world track it scores 0.461 versus 0.377 for the strongest baseline (DSPy), winning 5/7 columns, demonstrating that harness strategies transfer to unseen benchmarks without retraining on 8 of 12 target models. MoMHa attains the highest measured behavioral safety composite (U-SafeBench, 0.781) and uses 95 fewer tokens per example than the two-phase alternative. We will release all harness code, evaluation infrastructure, and crossmodel logs.