The Markovian Thinker
Free while signed in. Answers cite the passages they came from.
A new RL thinking environment that keeps an LLM’s effective state constant by chunking long chains of thought and carrying over only a short textual state between chunks. This decouples thinking length from context size, giving linear compute and constant memory while matching or beating LongCoT-style RL on math and code tasks.
Core idea: Reformulate the MDP: generate in fixed-size chunks of C tokens; at each boundary, reset the prompt to the original query plus the last m tokens from the previous chunk. The model learns to write a compact “Markovian state” near the end of each chunk to continue seamlessly after resets.
Why it matters for infra: For attention models, LongCoT training/inference scales quadratically with growing context. Delethink makes compute scale linearly with total thinking tokens and holds KV memory constant, because context never exceeds O(C).
Results at 24K budget (R1-Distill-1.5B): Trained with C=8K and m=C/2, Delethink matches/surpasses LongCoT-RL at the same 24K thinking budget on AIME’24/’25 and HMMT’25, and it maintains higher per-GPU rollout throughput because peak memory is flat.
Test-time scaling beyond train limit: Unlike LongCoT, which plateaus near its trained budget, Delethink keeps improving when you let it think longer at inference (e.g., up to 128K). Per-item plots show that certain AIME’25 questions only become solvable after very long traces.
Very long thinking with linear cost: Extending the iteration cap to I=23 enables a 96K budget with minimal extra training; average solutions reach 36–42K tokens while accuracy rises further. A cost projection estimates 27 H100-months for LongCoT-RL vs. 7 for Delethink at ~96K average thinking length.
Implementation notes: Training objective is a chunk-summed PPO/GRPO variant; pseudo-code for chunked rollouts is given. KV cache is cleared at chunk boundaries; the carryover is re-encoded, adding only a small prefill cost (p.6). Delethink is orthogonal to attention variants and could pair with sliding/streaming or SSMs inside chunks.
Zero-shot signal and generality: Off-the-shelf reasoning models (R1-Distill 1.5B–14B, Qwen3-30B-A3B, GPT-OSS-120B) already emit Markovian traces under Delethink tracing without training, often recovering most LongCoT performance and showing strong test-time scaling. Stress tests like CrossWordBench reveal limits when a large live state must be preserved.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack