Tiny Recursive Model

Core idea: Treat reasoning as repeated improvement. Given input x, current answer y, and latent z, the model performs n latent updates, then one answer update, for T recursions per supervision step. Unlike HRM, it backpropagates through a full recursion process and avoids the fixed-point one-step gradient approximation.

Tiny network, big gains: With ~7M params and self-attention, TRM hits 85.3% on Maze-Hard, 44.6% on ARC-AGI-1, and 7.8% on ARC-AGI-2, beating HRM’s 27M-param results of 74.5%, 40.3%, and 5.0. On Sudoku-Extreme, an attention-free MLP mixer variant reaches 87.4% vs HRM’s 55.0.

Design choices that matter: Single network replaces HRM’s two nets. Include x when updating z, exclude x when updating y to disambiguate roles. Ablations on page 5 show single-net > dual-net. Keep two features only. Interpreting y as the current decoded solution and z as latent reasoning works best; adding more z’s or collapsing to one hurts accuracy. Use attention only when L is large. For small, fixed grids like 9×9 Sudoku, a sequence-MLP outperforms attention; for 30×30 tasks (Maze, ARC), attention wins.

Efficient training loop: Deep supervision over up to 16 steps, a simpler halting head for ACT that avoids HRM’s extra forward pass, and EMA for stability on small data.

Single network replaces HRM’s two nets. Include x when updating z, exclude x when updating y to disambiguate roles. Ablations on page 5 show single-net > dual-net.

Keep two features only. Interpreting y as the current decoded solution and z as latent reasoning works best; adding more z’s or collapsing to one hurts accuracy.

Use attention only when L is large. For small, fixed grids like 9×9 Sudoku, a sequence-MLP outperforms attention; for 30×30 tasks (Maze, ARC), attention wins.