Stress Testing Large Reasoning Models
Proposes a new benchmark framework called REST to evaluate the robustness of Large Reasoning Models (LRMs) under multi-question stress. Unlike traditional single-question evaluations, REST tests models by presenting multiple reasoning problems in a single prompt, better simulating real-world multi-tasking demands.
Ask this paper
Stress reveals hidden weaknesses: Even top models like DeepSeek-R1 and Qwen-QwQ-32B, which score >95% on MATH500 in single-question settings, show large accuracy drops (e.g., -29% on AIME24 for DeepSeek-R1) when evaluated under REST. This challenges the assumption that LLMs are inherently capable of multi-problem reasoning.
Better discrimination among high performers: REST exposes significant performance gaps between models that perform similarly on single-question tasks. For instance, R1-7B and R1-32B differ by only 1.6% on MATH500 in single settings but diverge by over 22% under stress.
Long2Short training improves resilience: Models trained with concise reasoning objectives (e.g., L1-Qwen and Efficient-R1 variants) are significantly more robust under REST, preserving performance by avoiding the “overthinking trap.” For example, L1-Qwen-1.5B-Max maintains 73.2% accuracy on MATH500 under stress, outperforming similarly sized baselines.
Failure modes identified: Error analysis highlights frequent issues like question omission, reasoning errors, and uneven token allocation, particularly when earlier questions consume disproportionate reasoning effort. High-performing models adapt by balancing token usage across questions (“adaptive reasoning effort allocation”).