AI Papers of the Week
Every paper worth reading in AI, hand-picked one week at a time.
Step Back to Leap Forward
To boost the reasoning robustness of LLMs, researchers propose a “self-backtracking” mechanism that lets models revisit and revise their own intermediate reasoning steps. Key details:
Enhancing Reasoning to Adapt LLMs
Researchers from IBM present SOLOMON, a neuro-inspired LLM reasoning network architecture that boosts domain adaptability—demonstrated on semiconductor layout design. They show how LLMs often falter at spatial reasoning and domain knowledge application, and how their multi-agent oversight approach significantly improves success on challenging chip-layout tasks. Key insights include:
ReasonFlux
The ReasonFlux framework is introduced as an efficient way to fine-tune LLMs for complex reasoning, using hierarchical thought processes. Highlights include:
s1: Simple test-time scaling
Researchers from Stanford, UW, and others introduce s1, a method to boost LLM performance by using extra compute at inference (“test-time scaling”). Key ideas include:
LIMO: Less Is More for Reasoning
Can a handful of examples teach complex math reasoning to LLMs? This new LIMO paper challenges the notion that we need huge fine-tuning datasets for tough reasoning tasks. Key findings:
CoAT: Chain-of-Associated-Thoughts for LLM Reasoning
This work introduces CoAT, a new “slow thinking” inference framework that enables an LLM to reason more like a human by exploring and updating its thoughts. Main components:
Syntriever: Training Retrievers with LLM-Generated Data
How can we build a high-quality text retriever without large labeled datasets or access to an LLM’s internals? Syntriever presents a two-stage framework to distill knowledge from a black-box LLM into a retrieval model using synthetic data. Steps:
Demystifying Long Chain-of-Thought Reasoning in LLMs
This work investigates how LLMs develop extended CoT reasoning, focusing on RL and compute scaling. Key insights include:
MaAS: Multi-agent Architecture Search (Agentic Supernet)
Building multi-agent systems of LLMs (where multiple agents collaborate, each with specific roles or tools) is powerful but usually requires hand-designing a single complex pipeline. MaAS (Multi-agent Architecture Search) instead learns a universal “agentic supernet” from which it can spawn an optimal agent team on the fly for each query. It automates designing the agent workflow per task:
Advancing Reasoning in LLMs
This survey paper provides a timely overview of emerging methods to enhance reasoning capabilities in LLMs. It organizes the literature into several key approach categories:
o3-mini
OpenAI has launched o3-mini, their newest cost-efficient reasoning model, available in ChatGPT and API. The model excels in STEM-related tasks, particularly in science, math, and coding, while maintaining the low cost and reduced latency of its predecessor o1-mini. It introduces key developer features like function calling, Structured Outputs, and developer messages, making it production-ready from launch. o3-mini includes different reasoning effort levels (low, medium, and high) and improves performance across a wide range of tasks. It delivered responses 24% faster than o1-mini and achieved notable results in competition math, PhD-level science questions, and software engineering tasks.
On the Underthinking of o1-like LLMs
This work looks more closely at the "thinking" patterns of o1-like LLMs. We have seen a few recent papers pointing out the issues with overthinking. There is now a new phenomenon called underthinking! What is it about? The authors find that o1-like LLMs frequently switch between different reasoning thoughts without sufficiently exploring promising paths to reach a correct solution.
Usage Recommendation for DeepSeek-R1
This work provides a set of recommendations for how to prompt the DeepSeek-R1 model. Below are the key guidelines: 1. Prompt Engineering:
DeepSeek-R1
DeepSeek introduces DeepSeek-R1, an advancement in reasoning capabilities achieved through reinforcement learning (RL). It involves two key models: DeepSeek-R1-Zero, which uses pure RL without supervised fine-tuning, and DeepSeek-R1, which combines RL with cold-start data. DeepSeek-R1-Zero demonstrates that models can develop sophisticated reasoning abilities through RL alone, achieving a 71.0% pass rate on AIME 2024 and matching OpenAI-o1-0912's performance. During training, it naturally evolved complex behaviors like self-verification and reflection. However, it faced challenges with readability and language mixing. To address these limitations, DeepSeek-R1 uses a multi-stage approach: initial fine-tuning with high-quality chain-of-thought examples, reasoning-focused RL training, collecting new training data through rejection sampling, and final RL optimization across all scenarios. This resulted in performance comparable to OpenAI-o1-1217, with 79.8% accuracy on AIME 2024 and 97.3% on MATH-500, while maintaining output readability and consistency. DeepSeek also successfully distilled DeepSeek-R1's capabilities into smaller models, with their 7B model outperforming larger competitors and their 32B model achieving results close to OpenAI-o1-mini. This demonstrates the effectiveness of distilling reasoning patterns from larger models rather than training smaller models directly through RL.
Scaling RL with LLMs
Kimi introduces k1.5, a multimodal LLMtrained using RL that achieves state-of-the-art performance across reasoning tasks. The model leverages long context scaling up to 128k tokens and improved policy optimization methods, establishing a simplified yet effective RL framework without complex techniques like Monte Carlo tree search or value functions. Notably, k1.5 matches OpenAI's o1 performance on various benchmarks including 77.5 on AIME and 96.2 on MATH 500. The model also introduces effective long2short methods that use long-chain-of-thought techniques to improve shorter models, achieving superior results in constrained settings. Using these techniques, k1.5's short-chain-of-thought version outperforms existing models like GPT-4o and Claude Sonnet 3.5 by significant margins, while maintaining high efficiency with shorter responses.
Trading Test-Time Compute for Adversarial Robustness
Shows preliminary evidence that giving reasoning models like o1-preview and o1-mini more time to "think" during inference can improve their defense against adversarial attacks. Experiments covered various tasks, from basic math problems to image classification, showing that increasing inference-time compute often reduces the success rate of attacks to near zero. The approach doesn't work uniformly across all scenarios, particularly with certain StrongREJECT benchmark tests, and controlling how models use their compute time remains challenging. Despite these constraints, the findings suggest a promising direction for improving AI security without relying on traditional adversarial training methods.
Learning to Memorize at Test Time
introduces a neural long-term memory module to memorize historical context and help attention to attend to the current context while utilizing long past information; the neural memory module acts as a long-term, more persistent memory than just using attention alone (considered more short-term); Titan, which is based on neural memory, shows good results in language modeling, common-sense reasoning, genomics, and time series tasks.
Imagine while Reasoning in Space
introduces MVoT (Multimodal Visualization-of-Thought), a new reasoning framework that enables AI models to "think" in both text and images; MVoT enhances the traditional Chain-of-Thought prompting by allowing models to generate visual representations of their reasoning steps alongside text explanations; the framework is implemented in Chameleon-7B, a multimodal language model, and introduces a "token discrepancy loss" to improve the quality of generated visualizations; MVoT significantly outperforms traditional approaches, especially in complex scenarios; MVoT achieves over 90% accuracy on maze and printer installation tasks.
ChemAgent
presents a new framework designed to improve the performance of LLMs on chemical reasoning through a dynamic, self-updating library; the library is developed by decomposing chemical tasks into sub-tasks and compiling them into a structured collection that can be referenced for future queries; when the system is given a new problem, it retries and refines relevant information from the library to enable more effective task decomposition; the library is dynamically updated with new sub-tasks and solutions as they are encountered and validated; experiments on SciBench demonstrate that ChemAgent achieves performance gains of up to 46% (GPT-4), significantly outperforming existing methods.
Search-o1
a framework that combines large reasoning models (LRMs) with agentic search and document refinement capabilities to tackle knowledge insufficiency; the framework enables autonomous knowledge retrieval during reasoning and demonstrates strong performance across complex tasks, outperforming both baseline models and human experts.
Towards System 2 Reasoning
proposes Meta Chain-of-Thought (Meta-CoT), which extends traditional Chain-of-Thought (CoT) by modeling the underlying reasoning required to arrive at a particular CoT; the main argument is that CoT is naive and Meta-CoT gets closer to the cognitive process required for advanced problem-solving.
rStar-Math
a new approach proposes three core components to enhance math reasoning: 1) a code-augmented CoT data synthesis method involving MCTS to generate step-by-step verified reasoning trajectories which are used to train the policy SLM, 2) an SLM-based process reward model that reliably predicts a reward label for each math reasoning step, and 3) a self-evolution recipe where the policy SLM and PPM are iteratively evolved to improve math reasoning; on the MATH benchmark, rStar-Math improves Qwen2.5-Math-7B from 58.8% to 90.0% and Phi3-mini-3.8B from 41.4% to 86.4%, surpassing o1-preview by +4.5% and +0.9%.
Process Reinforcement through Implicit Rewards
a framework for online reinforcement learning that uses process rewards to improve language model reasoning; the proposed algorithm combines online prompt filtering, RLOO return/advantage estimation, PPO loss, and implicit process reward modeling online updates; on their model, Eurus-2-7B-PRIME, achieves 26.7% pass@1 on AIME 2024, surpassing GPT-4 and other models, using only 1/10 of the training data compared to similar models.
Machine-Assisted Proof
examines how mathematicians have long used machines to assist with mathematics research and discusses recent AI tools that are transforming mathematical proof assistance.