JevAgentHarness: Benchmarking, Deploying and Training Decision Models That Share Control with LLMs
Kunlun Zhu, Tao Feng and colleagues at UIUC (with Jiaxuan You and Heng Ji) test whether an agent harness can hand routine multiple-choice decisions to a fast decision model such as Jev, and build a harness and an open 4B model that do so with little loss of task success.
Ask this paper
Isolated scores misrank models. Across nine decision models, accuracy on isolated questions spans 5.1 points but task success spans 57.5 points. StartLux-9B was most accurate on BFCL tool choice (92.2%) yet solved only 69.5% of agent tasks; AgentDecision-4B scored 89.3% in isolation and solved 91.5%.
Fitted threshold. JevAgentHarness sends each choice to the decision model as a typed multiple-choice question and uses the answer only when confidence clears a threshold fitted on the agent's own trajectories. With Gemini 3.5 Flash as the agent, Jev made 27.2% of tool choices and 20.8% of argument choices, cutting cost by 14.2% without changing the outcome of any of 500 tasks.
Modes without a threshold fail. Letting Jev make every tool choice lost 19.0 points, and a fixed threshold of 0.5 lost 21.5 points. Of 40 point-mode combinations, the tool and argument cascades plus a reflection trigger were the delegations that passed.
AgentDecision-4B. An open 4B model trained with SFT and then GRPO inside the agent loop on states the agent visits after its own mistakes, with Jev's answers as labels. On shifted tasks the agent solved 91.5% with it against 82.0% with Jev; at must-ask states it answered 67.4% correctly against 58.3% for Jev.
Cost and limits. On customer-service and household tasks, 4B models trained on each agent's states made agents 81.1-89.0% cheaper and up to 2.5 times faster, within 3 points of success. Refitted on tau-bench, the cheapest passing delegation lost 8.1 points, so transfer is not shown.
Abstract
Large language model (LLM) agents now fix software, operate computers and serve customers, but their harnesses, the programs that run them, call a large LLM at every step, which makes long tasks slow and costly. Most of these calls end in a multiple-choice decision, such as which tool to call next. Decision models such as Jev answer these questions directly, at a fraction of an LLM call's time and price. We ask how a harness can delegate these decisions to a decision model with little loss of success. To evaluate decision models in agentic tasks, we build AgentDecisionBench, which tests them from isolated questions up to control of a running agent, on public benchmarks and our DecisionGym environment. The decision model most accurate on isolated questions solved only 69.5% of agent tasks, and without a fitted confidence threshold, delegating tool choices lowered success by 19.0 points. Building on these findings, we then design JevAgentHarness, which fits this threshold for each choice on the agent's own trajectories and lets the LLM decide below it. Because existing decision models never see the states an agent reaches after its own mistakes, we train AgentDecision-4B, an open 4B model, on such states with Jev's answers as labels. On tasks unlike its training data, the agent solved 91.5% of them with AgentDecision-4B and 82.0% with Jev. On tasks like their training data, 4B models trained this way made agents 81.1-89.0% cheaper and up to 2.5 times faster, without losing 3 or more points of task success. The models, data, and harness will be released at https://github.com/JevAgentHarness/JevAgentHarness.