When the harness decides success: state that cannot be re-observed
Raimondas Lencevicius (Microsoft) runs pre-registered experiments on when harness context choices, such as dropping, summarizing or replaying earlier turns, change whether an agent succeeds rather than only what it costs.
Ask this paper
Re-observable facts. In a synthetic probe, truncating context costs a reasoning model nothing when a needed fact can be looked up again, and drops it to chance when the fact cannot be re-observed. The two penalties differ by 92.5 points (90% CI 89.5 to 95.5), and a second provider's model shows the same contrast.
ARC-AGI-3 decomposition. On four games with one model, summarizing old turns while keeping the Standard harness's invitation to take notes recovers 86% of a 63-point gap to the Provider Adapter harness. Without the invitation it recovers 43%.
Reasoning replay. Adding the provider's native replay of the model's earlier reasoning recovers the remaining 57%. The author notes these are single-run point estimates per configuration.
Takeaway. Harness designers should identify which state cannot be fetched again and keep it, because losing that state is what changes success.
Abstract
Agent harnesses decide what a language model sees across turns: they drop old turns, summarize them, or replay the model's own earlier reasoning. We ask when these choices change whether an agent succeeds, not only what it costs, and report three findings. First, in a pre-registered synthetic probe, truncating the context window costs a low-cost reasoning model nothing in success when a needed fact can be looked up again (it looks the fact up instead), but drops its success to chance when the fact cannot be re-observed: the two penalties differ by 92.5 percentage points (90% CI [89.5, 95.5]). A pre-registered replication on one model from a second provider showed the same contrast. Once the fact was lost, that model chose no answer rather than guessing. Second, on ARC-AGI-3, where the benchmark's Standard harness trails its Provider Adapter harness by a wide margin, we decompose that gap one component at a time, for one model on four games. Summarizing old turns instead of dropping them, while keeping the Standard harness's invitation to take notes, recovers 86% of a 63-point gap without replaying the model's reasoning. The same harness without the invitation recovers 43%, and adding the provider's native replay of the model's reasoning recovers the remaining 57%. These are point estimates from one counted run per configuration and game, and the replay share depends on a single run. Third, in the benchmark's published runs and in ours, the Adapter's advantage in the probability of clearing a level it has reached is larger at deep levels than at early ones. This largely restates where runs stopped and does not separate a growing advantage from a constant one.