EMHO: EMbodied Agent Harness Optimization via Experience Traces
Hyun Jung Lee, Donghyun Kim (Korea University), Yong Jae Lee (UW-Madison) and colleagues present EMHO, which keeps an embodied VLM frozen and rewrites its harness from execution traces under sparse feedback.
Ask this paper
What gets optimized. The harness covers planning, context and memory, vision-tool use, observation grounding and failure recovery. Earlier work revised only skills or recovery prompts.
Loop. An inner loop runs EmbodiedBench tasks and records trajectories; an outer loop analyzes traces and prior harness versions and edits the harness code.
Results. With Qwen 9B and 27B models, task success improves by up to 23.3% on navigation and 33.3% on manipulation over the initial harness, and evolved harnesses also beat it on held-out tasks.
Behavior changes. After blocked moves, depth-estimation use rises from about 70% to nearly 100% and valid follow-up actions from about 44% to 68%. Wrong object selection in spatial-relation plans falls from 50% to 34%.
EMHO-Merge. Uses episode-level gains and losses to merge per-subtask revisions into one shared harness with fewer trade-offs.
Abstract
Improving embodied agents often focuses on optimizing the underlying model through training, while the surrounding agent harness that controls planning, context, and tool use is typically engineered. We ask whether this harness can instead improve itself directly from experience traces under sparse environmental feedback. We propose EMbodied Agent Harness Optimization (EMHO), a self-evolving framework that keeps the embodied model frozen and iteratively revises its harness by analyzing execution trajectories and prior harness history. EMHO optimizes beyond skills or recovery prompts, modifying how the agent monitors progress, uses vision tools, grounds observations, and responds to failures. To support multiple subtasks with a single harness, we introduce EMHO-Merge, which addresses trade-offs in jointly optimizing a single shared harness across subtasks by using episode-level gains and losses to guide evidence-supported refinement of when and how revised behaviors are applied. We evaluate EMHO on EmbodiedBench across navigation and manipulation tasks, and EMHO consistently improves task success for both Qwen 9B and 27B models. Qualitative analysis shows that EMHO goes beyond recovering from failures and unproductive actions to reshape how the embodied agent interprets and interacts with its environment.