Research map / Large language models / Reasoning and chain of thought
Teaching Language Models to Reason with Tools
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CoRT, a post-training framework that teaches large reasoning models to use code interpreters efficiently via hint-engineering data synthesis (30 human-annotated samples), SFT, rejection fine-tuning, and GRPO-based RL with a code-execution penalty.
Problem
Large reasoning models make errors and inefficiencies on complex mathematical computation, and naively integrating code interpreters causes conflicts between internal probabilistic reasoning and external deterministic outputs, leading to unproductive deliberation; training data for such code-integra
Approach
Hint-engineering inserts targeted hints at decision points in reasoning traces (when the model starts manual complex calculation, and to stop distrust of code output) rather than a single generic prompt hint, yielding short, efficient code-integrated trajectories from very few samples; RL is then applied to smaller models with a persistent Jupyter-like environment, output masking, and a code-failu
Key results
- Hint-Engineering-RFT-32B reaches 81.3% average accuracy across five benchmarks, with about 30-36% fewer tokens than DeepSeek-R1-Distill-Qwen-32B and similar accuracy to QwQ-32B at about half the tokens.
- Hint-Engineering-1.5B-RL reaches 56.4% average versus 48.1% for DeepSeek-R1-1.5B, with about 50-59% token reduction.
- Prompt-Hint-1.5B-RL achieves 58.3% average, the best among lightweight models in the table, outperforming DeepScaleR-1.5B-Preview (56.3%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply hint-engineering SFT and RFT | DeepSeek-R1-Distill-Qwen-32B · five mathematical benchmarks | average accuracy: 81.3% | Helped |
| Apply prompt-hint SFT | DeepSeek-R1-Distill-Qwen-32B · five mathematical benchmarks | average accuracy: 81.8% | Helped |
| Add RL after SFT | 1.5B tool-integrated models · five mathematical benchmarks | average accuracy: approximately 8% absolute gain | Helped |
| Add code execution reward with penalty=0.1 (ablation) | Prompt-Hint-1.5B RL · AIME24 | accuracy: 37.9% versus 43.1% | Helped |
| Add code execution reward with penalty=0.1 (ablation) | Hint-Engineering-1.5B RL · AIME24 | accuracy: 38.3% versus 41.0% | Helped |
| Increase code reward penalty from 0.1 to 0.5 (ablation) | 1.5B RL models · AIME24 | accuracy: 41.4% | Hurt |
| Increase RL training data from 1K to 5K or 20K samples (ablation) | 1.5B RL models · AIME24 | optimal accuracy | No clear effect |
| Train on hard queries (avg@8=1/8) instead of easy or uniform queries (ablation) | 1.5B RL models · AIME24 | peak accuracy | Helped |
| Change topic distribution from uniform to hard-query topics (ablation) | 1.5B RL models · AIME24 | optimal accuracy | No clear effect |
| Use SFT instead of the base model (ablation) | 1.5B Prompt-Hint and Hint-Engineering models · AIME24 and MATH500-Level5 | Pass@k upper bound | No clear effect |
| Add hint-engineering to guide early computation and trust code results | tool-integrated reasoning models · AIME24 | token usage: approximately 5000 tokens reduction | Helped |
| Use hint-engineering rather than Prompt-Hint | 32B tool-integrated models · AIME24 | token usage: 7K vs 14K | Helped |
Surprising findings
- Scaling RL data from 1K to 20K samples did not improve optimal performance, supporting a less-is-more view.
- Prompt-Hint-SFT-32B slightly outperforms Hint-Engineering-RFT-32B on average (81.8 vs 81.3), so the hint-engineering advantage is mainly in token efficiency.
Limitations
- Hint insertion points were placed manually by annotators; automated insertion with DeepSeek-V3 and R1 was judged insufficiently precise.
- RL could not be run on 32B models due to compute constraints, so 1.5B models were trained via distillation from 32B models.
- Accuracy gains for the 32B model are not statistically significant at the 0.05 level (p=0.055).
Experimental setup
Datasets: STILL-3 NuminaMath-1.5 AIME (pre-2024)
Benchmarks: AIME24 AIME25 AMC23 MATH500 OlympiadBench
Metrics: pass@1 (avg@16 / avg@4) token usage Pass@k
Baselines: DeepSeek-R1-Distill-Qwen-32B QwQ-32B o3 o4-mini STILL-3-TOOL-32B START-32B ReTool-R1-32B DeepScaleR-1.5B-Preview ToRL-1.5B Prompt-Hint-SFT-32B
Builds on: DeepSeek-R1 GRPO START STILL-3 ToRA
Closest papers by meaning
- Scaling Code-Assisted Chain-of-Thoughts and Instructions for Model Reasoning (NeurIPS 2025 · similarity 0.89)
- Exploring the Limits of Sub-Billion Language Model Reasoners with Open Training Recipes (ICLR 2026 · similarity 0.88)
- Token Assorted: Mixing Latent and Text Tokens for Improved Language Model Reasoning (ICML 2025 · similarity 0.88)
- JiuZhang3.0: Efficiently Improving Mathematical Reasoning by Training Small Data Synthesis Models (NeurIPS 2024 · similarity 0.88)
- Hint Tuning: Less Data Makes Better Reasoners (NeurIPS 2026 · similarity 0.87)
- Training Language Models to Reason Efficiently (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.