TRACE: Training Reasoning Agents for Causal Exploration with Synthesized Rewards
Rui Sun, Zhan Shi and Bing He (independent researchers) train diagnostic reasoning agents by sampling an intervention, injecting it into a simulator, and generating the observations it would produce, so the hidden intervention supplies an oracle label for a task where real ground truth would require expert investigation.
Ask this paper
Engineering the verification asymmetry: RLVR works where answers are cheap to check. Root-cause diagnosis is not, and often stays ambiguous after the fact. Simulating the cause forward gives an objective reward while the agent still has to work backward through noisy, confounded, distributed evidence.
The environment: TRACE is a digital-advertising diagnostic environment with 12 root causes and fine-grained segment attribution. Agents investigate with Python and SQL and must name both the root cause and the affected segment assignment.
Training beats prompting frontier models: On a 235-episode held-out test set the strongest prompted baseline, Claude Opus 5, reaches 0.686 FullAttr@1. SFT raises Qwen3.5-35B-A3B from 0.159 to 0.637 and subsequent RL with synthesized rewards reaches 0.757, above every prompted baseline including a prompted Qwen3.5-122B-A10B.
Fewer tool calls too: The trained policy uses substantially fewer tool calls than the prompted 35B base, so the gain is not bought with more investigation.
The claim the numbers support: Access to a scalable objective training signal constrains performance more than model scale does in this setting.
Abstract
Reinforcement learning with verifiable rewards (RLVR) has advanced language-model reasoning in domains such as mathematics and code, where objective answers are inexpensive to check. Diagnostic reasoning over complex data lacks this advantage: establishing the true cause of an anomaly often requires costly expert investigation and may remain ambiguous after the fact. We ask whether this asymmetry of verification can instead be engineered. We sample an intervention, inject it into a controlled simulator, and generate the observations it would produce. The hidden intervention provides an oracle label and objective reward, while the agent must still investigate noisy, confounded, and distributed evidence. We instantiate this approach in TRACE, a digital-advertising diagnostic environment with 12 root causes and fine-grained segment attribution. Agents investigate each episode using Python and SQL and must identify both the root cause and, when applicable, the affected segment assignment. On a held-out 235-episode test set, the strongest prompted baseline, Claude Opus 5, reaches 0.686 FullAttr@1. Supervised fine-tuning raises Qwen3.5-35B-A3B from 0.159 to 0.637, and subsequent RL with synthesized rewards reaches 0.757, outperforming all evaluated prompted baselines, including frontier closed-source models and a prompted Qwen3.5-122B-A10B model. The resulting policy also uses substantially fewer tool calls than the prompted 35B base. These results provide evidence that access to a scalable, objective training signal can be a more important constraint than model scale alone. More broadly, simulation-based verification can make otherwise ambiguous diagnostic reasoning tasks amenable to scalable reinforcement learning.