Research map / Large language models / Reasoning and chain of thought
Beyond Accuracy: Dissecting Mathematical Reasoning for LLMs Under Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SPARKLE, a three-axis analysis framework (plan following, knowledge integration, subproblem chains) and a 2,564-question augmented math benchmark, plus SparkleRL-PSS, a two-stage GRPO pipeline that reuses hard problems with partial step scaffolding.
Problem
Existing accuracy-only benchmarks give limited insight into what RL for LLM math reasoning actually improves, and hard problems that yield no RL reward are usually filtered out, wasting training signal.
Approach
Dissect RL effects along plan following, knowledge availability, and subproblem chains using an augmented benchmark, and reuse hard problems in a second RL stage by supplying partial reasoning chunks as scaffolding instead of discarding them or generating new data.
Key results
- SparkleRL-Stage 2-pss reaches 50.42% on AIME24 and 67.03% average over five benchmarks, versus 35.23% for the base Qwen-2.5-Math-7B and 65.01% for Stage 1.
- Human-crafted plans degrade the base model on most benchmarks, while RL-tuned models stay stable or improve slightly with plans (except AIME24).
- Average subproblem success on AIME24 is 17.5% for Stage 2-pss versus 50.4% full-problem accuracy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply Stage 1 GRPO training on math problems | Qwen-2.5-Math-7B · AIME24, AMC23, MATH500, GSM8K, OlympiadBench | Avg@8 accuracy: average of 29.78% improvement | Helped |
| Add partial solution scaffolding to Stage 2 hard-problem RL (ablation) | SparkleRL-Stage 1 · AIME24 | Avg@8 accuracy: 46.67 -> 50.42 | Helped |
| Add partial solution scaffolding to Stage 2 hard-problem RL (ablation) | SparkleRL-Stage 1 · AMC23 | Avg@8 accuracy: 67.50 -> 71.25 | Helped |
| Add partial solution scaffolding to Stage 2 hard-problem RL (ablation) | SparkleRL-Stage 1 · MATH500 | Avg@8 accuracy: 80.00 -> 81.00 | Helped |
| Add partial solution scaffolding to Stage 2 hard-problem RL (ablation) | SparkleRL-Stage 1 · GSM8K | Avg@8 accuracy: 91.77 -> 92.38 | Helped |
| Add partial solution scaffolding to Stage 2 hard-problem RL (ablation) | SparkleRL-Stage 1 · OlympiadBench | Avg@8 accuracy: 39.11 -> 40.11 | Helped |
| Continue RL on hard problems without partial scaffolding (ablation) | SparkleRL-Stage 1 · AIME24, AMC23, OlympiadBench | Avg@8 accuracy | Hurt |
| Continue RL on hard problems without partial scaffolding (ablation) | SparkleRL-Stage 1 · MATH500, GSM8K | Avg@8 accuracy | Helped |
| Continue RL on a mixture of easy and hard problems (ablation) | SparkleRL-Stage 1 · AIME24, AMC23, OlympiadBench | Avg@8 accuracy | Hurt |
| Continue RL on a mixture of easy and hard problems (ablation) | SparkleRL-Stage 1 · MATH500, GSM8K | Avg@8 accuracy | Helped |
| Provide supplementary external knowledge at inference | SparkleRL-Stage 1 · SPARKLE math benchmarks | accuracy: 4.3% | Helped |
| Provide supplementary external knowledge at inference | SparkleRL-Stage 2-pss · SPARKLE math benchmarks | accuracy: 4.2% | Helped |
Surprising findings
- Providing correct human-crafted step-by-step plans can degrade base-model performance on the hardest benchmarks.
- Supplying partial solution chunks is only beneficial when chunks are complete; partial initial steps may limit benefit.
Limitations
- Focused on structured math problems; generalization to less structured domains untested.
- Dataset construction depends on human annotation and may face scalability challenges.
- Findings are empirical with no theoretical characterization of internal strategies.
Experimental setup
Datasets: DeepScaleR-Preview NuminaMath AIME24 AMC23 MATH500 GSM8K OlympiadBench
Benchmarks: AIME24 AMC23 MATH500 GSM8K OlympiadBench
Metrics: Avg@8 accuracy pass@k subproblem success rate
Baselines: Qwen-2.5-Math-7B SparkleRL-Stage 1 SparkleRL-Stage 2-mix SparkleRL-Stage 2-hard
Builds on: GRPO DAPO DeepScaleR DeepSeek-R1
Closest papers by meaning
- RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs (ICLR 2026 · similarity 0.94)
- Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection (ICLR 2026 · similarity 0.94)
- Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspective (NeurIPS 2025 · similarity 0.93)
- On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models (ICML 2026 · similarity 0.93)
- Tricks or Traps? A Deep Dive into RL for LLM Reasoning (ICLR 2026 · similarity 0.93)
- Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? (NeurIPS 2025 · similarity 0.93)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.