Decoding Reasoning: Unraveling the Strengths and Limitations of Thought Models in the Face of Problem Complexity

Recent advancements in frontier language models have produced Large Reasoning Models (LRMs) that emphasize detailed reasoning processes. While LRMs show enhanced performance on reasoning tasks, their core abilities, scaling behavior, and limitations are not fully understood. Traditional evaluations focus on mathematical and coding benchmarks primarily assessing final answer accuracy, often falling prey to data contamination. This research examines these shortcomings through controllable puzzle environments, permitting manipulation of complexity while retaining logical consistency. Findings reveal that LRMs suffer significant accuracy declines at higher complexities, demonstrating a counterintuitive trend where reasoning efforts initially increase but then drop despite sufficient resources. Performance is categorized into three regimes: 1) low-complexity tasks favor standard models, 2) medium complexity favors LRMs, and 3) both models collapse under high complexity. Notably, LRMs struggle with exact computation, lack consistent reasoning, and demonstrate limited understanding in their approach, prompting further inquiry into their reasoning capabilities.

Source link