Chronicle: Cut-Point Replay for Regression Testing of LLM Agents
Tisha Chawla and Susheem Koul at Microsoft present Chronicle, which records an agent run at its non-deterministic boundaries as immutable envelopes and replays a chosen subset of them while running the rest live, turning a recorded incident into a CI regression test.
Ask this paper
Cut-point replay is the operation. Serving some boundaries from the record and executing the complementary subset against new code is what lets a recorded failure test a code change, which trace-and-score tooling cannot do.
Recording costs 23 microseconds per crossing. That is 0.008 percent of an assumed 300 ms model call, so recording can stay on.
Full replay issues zero model calls and is bit-stable. Verified across 20 repetitions, which is the property that makes the test deterministic enough for CI.
Cut-point tests discriminate correctly on all 6 incidents. They fail on faulty code and pass on both guarded and benign changes, which is the requirement a regression test has to meet.
Stubbing every boundary catches nothing. In a mutation study of guarded tools, cut-point tests caught every mutant that let the recorded unsafe action through while the full-stub baseline using the same assertion caught none.
Abstract
Large language model responses are non-deterministic, so failures in LLM agents are hard to reproduce: a failure depends on inference that is not bitwise reproducible, on tools that read changing state, and on a multi-step trajectory that a re-run rarely repeats. Record-and-replay makes a run reproducible, but existing agent tooling records runs only to trace or score them, not to test a code change against them. We present Chronicle, which records an agent run at its non-deterministic boundaries as immutable envelopes and replays it from the record. Its central operation, cut-point replay, serves a chosen subset of boundaries from the record and executes the complementary subset live with new code, turning a recorded incident into a regression test that runs in continuous integration. On a benchmark of 6 recorded failures with simulated model boundaries, recording adds 23 μs per crossing (0.008% of an assumed 300 ms model call), full replay issues zero model calls and is bit-stable across 20 repetitions, and cut-point tests fail on faulty code and pass on guarded and benign changes for all 6 incidents. In a mutation study of the guarded tools, cut-point tests catch every mutant that lets the recorded unsafe action through, while a baseline that stubs every boundary, using the same assertion, catches none. Chronicle and the benchmark are publicly available at https://github.com/theagentplane/chronicle.