Universal Reasoning Model
Free while signed in. Answers cite the passages they came from.
This paper investigates why universal transformers excel at complex reasoning tasks like ARC-AGI. The key finding: performance gains come primarily from recurrent inductive bias and strong nonlinear components rather than elaborate architectural designs. - **Recurrent mechanism matters most:** Through extensive ablation studies, the researchers show that reasoning capability beyond standard transformers comes from the recurrent mechanism of universal transformers, not from overly elaborate designs in prior work. - **ConvSwiGLU enhancement:** The Universal Reasoning Model (URM) augments the standard SwiGLU feed-forward block with depthwise short convolution, injecting local contextual interactions into the gating mechanism without increasing sequence-level complexity. - **Truncated backpropagation:** The approach uses truncated backpropagation through loops, enabling efficient training of the recurrent architecture while maintaining strong performance on reasoning tasks. - **State-of-the-art ARC-AGI results:** URM achieves 53.8% pass@1 on ARC-AGI 1 and 16.0% pass@1 on ARC-AGI 2, substantially outperforming prior UT-based models like TRM (40%) and HRM (32%) on ARC-AGI 1.
Recurrent mechanism matters most: Through extensive ablation studies, the researchers show that reasoning capability beyond standard transformers comes from the recurrent mechanism of universal transformers, not from overly elaborate designs in prior work.
ConvSwiGLU enhancement: The Universal Reasoning Model (URM) augments the standard SwiGLU feed-forward block with depthwise short convolution, injecting local contextual interactions into the gating mechanism without increasing sequence-level complexity.
Truncated backpropagation: The approach uses truncated backpropagation through loops, enabling efficient training of the recurrent architecture while maintaining strong performance on reasoning tasks.
State-of-the-art ARC-AGI results: URM achieves 53.8% pass@1 on ARC-AGI 1 and 16.0% pass@1 on ARC-AGI 2, substantially outperforming prior UT-based models like TRM (40%) and HRM (32%) on ARC-AGI 1.
Abstract
Universal transformers (UTs) have been widely used for complex reasoning tasks such as ARC-AGI and Sudoku, yet the specific sources of their performance gains remain underexplored. In this work, we systematically analyze UTs variants and show that improvements on ARC-AGI primarily arise from the recurrent inductive bias and strong nonlinear components of Transformer, rather than from elaborate architectural designs. Motivated by this finding, we propose the Universal Reasoning Model (URM), which enhances the UT with short convolution and truncated backpropagation. Our approach substantially improves reasoning performance, achieving state-of-the-art 53.8% pass@1 on ARC-AGI 1 and 16.0% pass@1 on ARC-AGI 2. Our code is avaliable at this https URL.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack