Fine-tuning LLM Agents without Fine-tuning LLMs
A memory‑based learning framework that lets deep‑research agents adapt online without updating model weights. The agent is cast as a memory‑augmented MDP with case‑based reasoning, implemented in a planner–executor loop over MCP tools. It sets top validation results on GAIA and delivers strong scores on DeepResearcher, SimpleQA, and HLE.
Ask this paper
Method in a line: Decisions are guided by a learned case‑retrieval policy over an episodic Case Bank. Non‑parametric memory retrieves Top‑K similar cases; parametric memory learns a Q‑function (soft Q‑learning or single‑step CE training in deep‑research settings) to rank cases for reuse and revision.
Architecture: Planner (LLM CBR) + Executor (LLM MCP client) with three memories: Case, Subtask, Tool. Involves a loop for planning, tool execution, writing/reading of cases, and a replay buffer. Tools span search, crawl, multimodal document parsing, code execution, and math utilities.
Results: • GAIA: 87.88% Pass@3 on validation and 79.40% on test, competitive with or above open‑source agent frameworks. • DeepResearcher: 66.6 F1 and 80.4 PM average across seven open‑domain QA sets. • SimpleQA: 95.0% accuracy, beating recent web‑agent baselines. • HLE: 24.4 PM, close to GPT‑5 and ahead of several strong baselines.
Ablations and scaling: • Case count: performance peaks around K = 4 retrieved cases, emphasizing small, high‑quality memory rather than many shots. • Continual learning: both non‑parametric and parametric CBR yield steady gains over iterations vs. no‑CBR. • Component study: moving from offline to online tools helps, adding planning helps more, and adding CBR yields the largest consistent boost across benchmarks. • Cost profile: input tokens, not outputs, drive costs as difficulty rises.
Practical takeaways for agent builders: • Use a compact, curated case memory with adaptive retrieval rather than growing prompts. • Keep planning concise. A fast planner outperforms slow‑think planners for multi‑step tool use on GAIA by avoiding verbose or shortcut plans. • Separate planning and execution with explicit Subtask and Tool memories to coordinate long‑horizon work and reduce hallucinations.