Research map / Large language models / Agents, code and math
Learning Theorem Rationale for Improving the Mathematical Reasoning Capability of Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Theorem Rationale (TR): generates question-theorem-solution triples with GPT-4o and evolves theorem-oriented instructions (TM, TA, TPS) to fine-tune LLaMA3-8B for math reasoning.
Problem
LLMs struggle with advanced high-school and college-level math because they do not explicitly select and apply the relevant mathematical theorems, leading to missing or misapplied theorems and hallucination.
Approach
Teach the model to explicitly list relevant theorems before solving (theorem rationale), building triples from GPT-4o reasoning paths and expanding them with three theorem-centred instruction types (memorization, alignment, theorem-based problem solving) to fine-tune an open 8B model.
Key results
- The TR-tuned LLaMA3-8B averages 33.1 across seven datasets versus 29.5 for the LLaMA-3 backbone, with GSM8k rising from 74.2 to 78.9 and JEEBench from 6.8 to 8.9.
- Training on TR-only data (30k) reaches 74.7 on GSM8k and 13.2 on JEEBench, outperforming the mixed setting on college-level benchmarks with less data.
- Removing the theorem-based problem solving (TPS) subset causes the largest accuracy drop in ablation, while removing theorem memorization (TM) has the smallest effect.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune LLaMA3-8B on mixed TR and CoT data | LLaMA3-8B · SAT-Math | accuracy: 54.1 -> 61.8 | Helped |
| Fine-tune LLaMA3-8B on mixed TR and CoT data | LLaMA3-8B · SciBench | accuracy: 6.9 -> 7.7 | Helped |
| Fine-tune LLaMA3-8B on mixed TR and CoT data | LLaMA3-8B · JEEBench | accuracy: 6.8 -> 8.9 | Helped |
| Fine-tune LLaMA3-8B on mixed TR and CoT data | LLaMA3-8B · MMLU-pro-Math | accuracy: 33.9 -> 36.9 | Helped |
| Fine-tune LLaMA3-8B on mixed TR and CoT data | LLaMA3-8B · MATH | accuracy: 27.1 -> 28.2 | Helped |
| Fine-tune LLaMA3-8B on mixed TR and CoT data | LLaMA3-8B · GSM8k | accuracy: 74.2 -> 78.9 | Helped |
| Fine-tune LLaMA3-8B on mixed TR and CoT data | LLaMA3-8B · Conic10K | accuracy: 3.4 -> 9.6 | Helped |
| Use TR-only training data instead of CoT-only data (ablation) | LLaMA3-8B · GSM8k | accuracy: 61.1 -> 74.7 | Helped |
| Use TR-only training data instead of CoT-only data (ablation) | LLaMA3-8B · MMLU-pro-Math | accuracy: 26.6 -> 35.6 | Helped |
| Use TR-only training data instead of CoT-only data (ablation) | LLaMA3-8B · JEEBench | accuracy: 2.5 -> 13.2 | Helped |
| Use TR-only training data instead of CoT-only data (ablation) | LLaMA3-8B · SciBench | accuracy: 1.6 -> 10.3 | Helped |
| Use TR-only training data instead of CoT-only data (ablation) | LLaMA3-8B · SAT-Math | accuracy: 44.5 -> 52.3 | Helped |
Surprising findings
- TR-only training outperforms the mixed TR+CoT data on college-level benchmarks such as JEEBench and SciBench despite using less data.
- In error analysis, 76% of errors were incorrect theorem application or computation rather than wrong theorem selection.
Limitations
- Performance still lags closed-source models such as GPT-4 on advanced benchmarks.
- Most errors (76%) stem from theorem application and computation rather than theorem selection.
- Theorem feedback for error correction is currently provided by humans only.
Experimental setup
Datasets: GSM8k MATH OlympiadBench College-Math Conic10k TheoremQA
Benchmarks: GSM8k MATH Conic10K MMLU-pro-Math JEEBench-Math SciBench SAT-Math
Metrics: accuracy
Baselines: LLaMA-3 LLaMA-2 Galactica-6.7B AQuA-SFT WizardMath MAmmoTH-CoT GPT-4 GPT-3.5 Claude-2
Builds on: chain-of-thought prompting MAmmoTH WizardMath
Closest papers by meaning
- Advancing Mathematical Reasoning in Language Models: The Impact of Problem-Solving Data, Data Synthesis Methods, and Training Stages (ICLR 2025 · similarity 0.90)
- Evaluating Mathematical Reasoning Beyond Accuracy (AAAI 2025 · similarity 0.90)
- UGMathBench: A Diverse and Dynamic Benchmark for Undergraduate-Level Mathematical Reasoning with Large Language Models (ICLR 2025 · similarity 0.90)
- MathFimer: Enhancing Mathematical Reasoning by Expanding Reasoning Steps through Fill-in-the-Middle Task (ICLR 2026 · similarity 0.90)
- One Example Shown, Many Concepts Known! Counterexample-Driven Conceptual Reasoning in Mathematical LLMs (ICML 2025 · similarity 0.90)
- Neuro-Symbolic Data Generation for Math Reasoning (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.