Demystifying RL in Agentic Reasoning

Data > synthetic. Real, end-to-end multi-turn trajectories for SFT give a much stronger cold-start than stitched synthetic traces. On AIME24/25, real SFT boosts average@32 and pass@32 by large margins for 4B and 7B bases.

Diversity sustains exploration: A diversified RL dataset across math, science, and code raises and maintains policy entropy, speeding learning and stabilizing training. The model-aware curation further fixes weak-model bottlenecks by matching task difficulty to capability.

Simple GRPO tweaks matter: A practical recipe using token-level aggregation, higher clip range, and overlong-penalty shaping (GRPO-TCR) consistently outperforms a standard GRPO baseline in both peak accuracy and data efficiency.

Entropy needs a sweet spot: Training is best when policy entropy is neither collapsed nor excessive. Increasing the clip upper bound modestly accelerates progress, but too high degrades convergence and stability.

Deliberate mode wins: Fewer, better tool calls after more internal planning lead to higher tool-use success and overall accuracy than reactive short-think with frequent calls.

Long-CoT is not plug-and-play for agents: Off-the-shelf Long-CoT models avoid tools on reasoning-heavy tasks, driving tool-call counts toward zero during RL. SFT with multi-turn tool traces can re-align them, but instruction-tuned bases ultimately scale agentic capability more cleanly.

Compact SOTA with the recipe: Using the 30k diverse RL set and GRPO-TCR with a tuned clip upper bound, DemyAgent-4B reaches or beats much larger models in agentic settings, including AIME25, GPQA-Diamond, and LiveCodeBench-v6.