Eliciting Reasoning with Cognitive Tools
Free while signed in. Answers cite the passages they came from.
Proposes a modular, tool-based approach to eliciting reasoning in LLMs, inspired by cognitive science. Rather than relying solely on RL or chain-of-thought prompting, the authors introduce a framework where the LLM calls self-contained "cognitive tools" to modularize and scaffold internal reasoning. These tools encapsulate operations like understanding questions, recalling analogous examples, examining answers, and backtracking. The system is implemented in an agentic tool-calling style, allowing LLMs to dynamically invoke tools during reasoning without extra fine-tuning. Highlights:
Cognitive tools as internal modules: Each tool (e.g., understand question, recall related, examine answer, backtracking) is framed as a standalone prompt template that the LLM can invoke as needed. Unlike conventional tool use (e.g., calculator APIs), these tools operate within the LLM’s own architecture and memory.
Consistent performance gains: On math-heavy reasoning benchmarks like AIME 2024, MATH500, and AMC, the cognitive tools pipeline significantly boosts pass@1 accuracy across models, including Qwen2.5, Llama3, and GPT-4.1. For instance, Llama3.3-70B improves from 13.1% to 29.8% on AIME2024, and GPT-4.1 rises from 26.7% to 43.3%, nearly matching the o1-preview RL-trained reasoning model at 44.6%.
Superior to cognitive prompting: Compared to prior work on cognitive prompting, the modular tool approach shows stronger generalization and reduced reasoning interference. Tools can be invoked flexibly, and each invocation operates in a clean context window, boosting accuracy by up to +27.2% over baseline on Smolbenchmark.
Interpretable and transferable: The modular nature of the tools enhances transparency, and their plug-and-play design allows transfer across models and benchmarks with minimal changes. The approach also supports interpretability by surfacing intermediate reasoning steps and decisions.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack