CordisBench: Can Language Models Reason About Component Lifecycles in Dynamic Agent Harnesses?
Damien Sileo and Dimitri Kachler introduce CordisBench, a 1,200-question benchmark for a reasoning burden that only appears once agents can rewrite the software running them: predicting how a local component change propagates through dependencies and teardown.
Ask this paper
A new failure surface. Dynamic harnesses let a model change the software that shapes its own execution, so a plugin edit can cascade through dependency and cleanup logic in ways the model must anticipate before acting.
Four task types against a real runtime. Models identify affected components, predict state after a specified teardown order, determine which conditions hold under all or some orders, and choose reconfigurations that succeed when actually executed against Cordis.
Reliability degrades with interaction count. Across 2 to 32 relevant interactions, models handle small systems well and get less reliable as more interactions matter, worst on final-state prediction and cross-teardown-order reasoning.
The cost result is the practical one. On the 16-interaction subset, GPT-5.6 Luna spends nearly 3,000 reasoning tokens per question at medium effort, and the authors show that for these controlled instances the cost is avoidable: an independent finite reference semantics matches Cordis execution on every scored observation and action outcome across all 528 executable questions.
Read it as a warning about self-modifying agents. If lifecycle reasoning is this expensive and this brittle, letting an agent restructure its own harness is a much larger ask than letting it write application code.
Abstract
Dynamic agent harnesses let language models change the software that shapes their own execution. This flexibility brings a new reasoning burden: a local plugin change can propagate through dependencies and cleanup. We introduce CordisBench, a 1,200-question benchmark of this lifecycle reasoning. It combines a controlled formal setting with programs executed against Cordis, a runtime that manages component dependencies and cleanup, and asks models to identify affected components, predict state after a specified teardown order, determine which conditions hold under all or some orders, and choose reconfigurations that succeed when executed. Across these tasks, we evaluate three efficiency-oriented models at low reasoning effort with 2, 4, 8, 16, 24, or 32 relevant interactions, using deterministic task-specific scoring. Models usually handle small systems well but grow less reliable as more interactions become relevant, especially when predicting final state and when reasoning across teardown orders. Additional inference effort recovers marked gains for some models. The cost is nontrivial: on our 16-interaction subset, GPT-5.6 Luna uses nearly 3,000 reasoning tokens per question at medium effort. For these controlled instances, that cost is avoidable: an independent finite reference semantics agrees with Cordis execution on every observation and action outcome used for scoring across all 528 executable questions.