Research map / Large language models / Reasoning and chain of thought
Eliciting Reasoning in Language Models with Cognitive Tools
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes "cognitive tools" that wrap modular reasoning operations (understand question, recall related, examine answer, backtracking, code) as self-invoked LLM sub-calls within a tool-calling loop, eliciting math reasoning without RL.
Problem
Whether LLM reasoning can be elicited through alternative, non-RL methods, and whether modular cognitive operations offer gains over monolithic cognitive prompting or chain-of-thought on math benchmarks.
Approach
Encapsulates cognitive operations as discrete prompt-driven tools that the same LLM invokes within an agentic tool-calling loop, each executed in a sandboxed LLM call whose output feeds back into the main reasoning trace, rather than a fixed monolithic prompt.
Key results
- GPT-4.1 with cognitive tools raises AIME 2024 pass@1 from 32.0% to 53.0%, above o1-preview at 44.6%.
- Qwen2.5-32B Instruct with cognitive tools reaches 32.1% on AIME 2024 vs 17.2% baseline and 58.9 average across three benchmarks vs 48.0 baseline.
- Cognitive tools outperform cognitive prompting on Smolbenchmark across all four open models, e.g. 80.0 vs 66.0 on Llama3.3-70B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add cognitive tools | Qwen2.5-7B Instruct · AIME 2024, MATH500, AMC | pass@1 accuracy: 12.5 $\pm$ 0.7 -> 14.6 $\pm$ 1.8; 71.7 $\pm$ 1.3 -> 73.7 $\pm$ 0.5; 43.9 $\pm$ 1.3 -> 47.0 $\pm$ 0.5 | Helped |
| Add cognitive tools | Qwen2.5-32B Instruct · AIME 2024, MATH500, AMC | pass@1 accuracy: 17.2 $\pm$ 1.2 -> 32.1 $\pm$ 1.9; 74.1 $\pm$ 0.7 -> 81.8 $\pm$ 0.6; 52.6 $\pm$ 0.8 -> 62.7 $\pm$ 1.2 | Helped |
| Add cognitive tools | Llama3.1-8B Instruct · AIME 2024, MATH500, AMC | pass@1 accuracy: 5.8 $\pm$ 1.0 -> 8.8 $\pm$ 1.7; 43.2 $\pm$ 0.5 -> 50.7 $\pm$ 1.0; 20.3 $\pm$ 0.8 -> 28.0 $\pm$ 1.2 | Mixed |
| Add cognitive tools | Llama3.3-70B Instruct · AIME 2024, MATH500, AMC | pass@1 accuracy: 13.1 $\pm$ 1.0 -> 29.8 $\pm$ 1.2; 57.0 $\pm$ 0.5 -> 74.7 $\pm$ 0.5; 33.0 $\pm$ 0.9 -> 51.0 $\pm$ 0.5 | Helped |
| Add cognitive tools | GPT-4.1 · AIME 2024 | pass@1 accuracy: 32.0 -> 53.0 | Helped |
| Use cognitive tools instead of cognitive prompting | Qwen2.5-7B, Qwen2.5-32B, Llama3.1-8B, and Llama3.3-70B · Smolbenchmark | pass@1 accuracy: 75.8 vs 80.0; 79.6 vs 88.0; 48.9 vs 60.0; 52.8 vs 80.0 | Helped |
| Add understand question tool (ablation) | Llama3.3-70B Instruct · Smolbenchmark | pass@1 accuracy: +26.7% | Helped |
| Add individual cognitive tools (ablation) | Qwen2.5-7B and Qwen2.5-32B · Smolbenchmark | pass@1 accuracy: 75.8 -> 78.6, 76.1, 77.8, 80.5; 79.6 -> 82.5, 84.2, 84.0, 82.9 | Helped |
| Add individual cognitive tools (ablation) | Llama3.1-8B and Llama3.3-70B · Smolbenchmark | pass@1 accuracy | Helped |
| Remove motivational cues from the cognitive tools prompt (ablation) | Cognitive tools on the tested Llama and Qwen models · AIME2024 | pass@1 accuracy: no motivational cues: 16.7 $\pm$ 3.3, 32.1 $\pm$ 2.9, 9.8 $\pm$ 3.7, 36.7 $\pm$ 3.5; original: 14.6 $\pm$ 1.8, 32.1 $\pm | No clear effect |
| Use cognitive prompting instead of the plain baseline | Qwen2.5-7B Instruct · Smolbenchmark | pass@1 accuracy: 75.8 -> 74.0 | Hurt |
| Use cognitive prompting instead of the plain baseline | Llama3.1-8B Instruct · Smolbenchmark | pass@1 accuracy: 48.9 -> 47.1 | Hurt |
Surprising findings
- Removing motivational cues from the system prompt (e.g., the $1,000,000 reward line) did not hurt and slightly improved average AIME performance across models.
- Tool usage differs sharply by model family: Qwen rarely calls understand-question or examine-answer on AIME, while Llama calls them most of the time.
Limitations
- Cognitive tools are manually defined and prompts may need re-engineering for other model families.
- Evaluated only on math benchmarks.
- Roughly doubles output tokens (4,200 vs 2,000 on AIME for GPT-4.1).
Experimental setup
Datasets: Smolagents Benchmark-v1 math subset
Benchmarks: AIME 2024 MATH500 AMC Smolbenchmark
Metrics: pass@1 accuracy
Baselines: chain-of-thought cognitive prompting code-equipped baseline o1-preview
Builds on: cognitive prompting chain-of-thought Toolformer budget forcing ACT-R
Closest papers by meaning
- RAST: Reasoning Activation in LLMs via Small-model Transfer (NeurIPS 2025 · similarity 0.90)
- RL of Thoughts: Navigating LLM Reasoning with Inference-time Reinforcement Learning (ICLR 2026 · similarity 0.90)
- Nemotron-Research-Tool-N1: Exploring Tool-Using Language Models with Reinforced Reasoning (ICLR 2026 · similarity 0.90)
- On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models (ICML 2026 · similarity 0.90)
- Distilling LLM Agent into Small Models with Retrieval and Code Tools (NeurIPS 2025 · similarity 0.90)
- RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs (ICLR 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.