InftyThink+
Free while signed in. Answers cite the passages they came from.
InftyThink+ is an end-to-end RL framework for infinite-horizon reasoning that optimizes the entire iterative reasoning trajectory. Standard long chain-of-thought suffers from quadratic cost, context length limits, and lost-in-the-middle degradation. InftyThink+ addresses all three by letting models autonomously decide when to summarize, what to preserve, and how to resume, trained through trajectory-level reinforcement learning. - **Iterative reasoning with learned boundaries:** Instead of generating one continuous chain-of-thought, InftyThink+ decomposes reasoning into multiple iterations connected by self-generated summaries. The model learns to control iteration boundaries, deciding when to compress and continue rather than following fixed heuristics or chunk sizes. - **Two-stage training recipe:** A supervised cold-start teaches the InftyThink format (special tokens for summary and history), then trajectory-level GRPO optimizes the full multi-iteration rollout. Advantages are shared across all iterations within a trajectory, so early high-quality summaries that enable correct later reasoning receive positive gradient signal. - **21% accuracy gain on AIME24:** On DeepSeek-R1-Distill-Qwen-1.5B, InftyThink+ with RL improves accuracy from 29.5% to 50.9% on AIME24, a 21-point jump that substantially outperforms vanilla long-CoT RL (38.8%). Results generalize to out-of-distribution benchmarks including GPQA Diamond and AIME25. - **Faster inference, faster training:** By bounding context length per iteration, InftyThink+ reduces inference latency compared to vanilla reasoning while achieving higher accuracy. Adding an efficiency reward further cuts token usage by 50% with only modest accuracy trade-off, demonstrating a controllable speed-accuracy knob.
Iterative reasoning with learned boundaries: Instead of generating one continuous chain-of-thought, InftyThink+ decomposes reasoning into multiple iterations connected by self-generated summaries. The model learns to control iteration boundaries, deciding when to compress and continue rather than following fixed heuristics or chunk sizes.
Two-stage training recipe: A supervised cold-start teaches the InftyThink format (special tokens for summary and history), then trajectory-level GRPO optimizes the full multi-iteration rollout. Advantages are shared across all iterations within a trajectory, so early high-quality summaries that enable correct later reasoning receive a positive gradient signal.
21% accuracy gain on AIME24: On DeepSeek-R1-Distill-Qwen-1.5B, InftyThink+ with RL improves accuracy from 29.5% to 50.9% on AIME24, a 21-point jump that substantially outperforms vanilla long-CoT RL (38.8%). Results generalize to out-of-distribution benchmarks, including GPQA Diamond and AIME25.
Faster inference, faster training: By bounding context length per iteration, InftyThink+ reduces inference latency compared to vanilla reasoning while achieving higher accuracy. Adding an efficiency reward further cuts token usage by 50% with only a modest accuracy trade-off, demonstrating a controllable speed-accuracy knob.
Abstract
Large reasoning models achieve strong performance by scaling inference-time chain-of-thought, but this paradigm suffers from quadratic cost, context length limits, and degraded reasoning due to lost-in-the-middle effects. Iterative reasoning mitigates these issues by periodically summarizing intermediate thoughts, yet existing methods rely on supervised learning or fixed heuristics and fail to optimize when to summarize, what to preserve, and how to resume reasoning. We propose InftyThink+, an end-to-end reinforcement learning framework that optimizes the entire iterative reasoning trajectory, building on model-controlled iteration boundaries and explicit summarization. InftyThink+ adopts a two-stage training scheme with supervised cold-start followed by trajectory-level reinforcement learning, enabling the model to learn strategic summarization and continuation decisions. Experiments on DeepSeek-R1-Distill-Qwen-1.5B show that InftyThink+ improves accuracy by 21% on AIME24 and outperforms conventional long chain-of-thought reinforcement learning by a clear margin, while also generalizing better to out-of-distribution benchmarks. Moreover, InftyThink+ significantly reduces inference latency and accelerates reinforcement learning training, demonstrating improved reasoning efficiency alongside stronger performance.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack