Harness-Level Forgetting
Free while signed in. Answers cite the passages they came from.
Continual learning has always tracked what changes in the weights. Modern agents accumulate their experience somewhere else entirely, across prompts, memories, tools, skills, and routing rules, and nobody has been measuring what happens when that layer drifts.
The failure has a name now: Update any harness component and previously reliable behavior can break with the model completely untouched. The paper calls this harness-level forgetting and gives it a measurement protocol, which is what was missing.
Proposing and committing are different jobs: Guarded harness evolution splits them. A Continual Optimizer drafts a candidate harness from post-execution feedback, and a Continual Evaluator commits only after checking current improvement, historical retention, and validity.
It generalizes across modality: Relative gains exceed 10% across textual reasoning, multimodal perception, and open-world interaction, so this is not tuned to one benchmark family.
Why it matters: If you already let your agents rewrite their own prompts, skills, or memory files, you are running an unguarded version of this loop today. The gate between draft and commit is the cheap part to add, and the historical retention check is the one most self-editing setups skip.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack