What Else Needs Fixing? Exploring Cost-Effective Test-Time Compute for Revision Propagation in Artifacts Generated Through Conversation
Daisuke Kikuta (NTT) studies revision propagation, where a user asks for one local change and the model must find and update every dependent part of an artifact whose dependencies are buried in the conversation history.
Ask this paper
A new setting, not a new model: the artifact and its dependency structure live in the conversation itself, so the model must recover the graph before it can propagate anything.
Nine revision methods compared: sequential reflection and parallel sampling variants across gpt-oss-20b/120b, gpt-5.4-mini and qwen3.5-9b/27b/122b.
Baselines already reach 68.3 to 93%, which sets a high bar and leaves limited headroom for the methods being compared.
Three samples is the sweet spot: selecting from three parallel samples with either LLM-based or medoid selection is the most cost-effective method, adding 2.2 to 9.7% accuracy.
Why it matters: conversational artifact editing is what most people actually do with an assistant, and this is one of the few papers that measures the dependency-propagation half of it.
Abstract
Large Language Models (LLMs) often help users generate artifacts through iterative cycles of generation and revision in conversation. A challenge here is that, when users specify only a local change during revision, LLMs must instead identify the relevant dependencies and propagate the revision to all affected parts of the artifact. This paper studies this ability of LLMs on conversationally generated artifacts, where the artifact context and its dependencies may be embedded in the conversation history. Toward practical use, we also explore cost-effective test-time compute for this new setting. Specifically, we introduce a new benchmark for this setting, and evaluate nine revision methods, including sequential reflection and parallel sampling variants, using gpt-oss-20b/120b, gpt-5.4-mini, and qwen3.5-9b/27b/122b on the benchmark. The results show that baselines achieve accuracies of 68.3--93%, and the most cost-effective method is selecting from three parallel samples using either LLM-based or medoid selection, which improves accuracy by 2.2--9.7%. Our code and dataset are available at https://github.com/ntt-dkiku/llm-revision-propagation.