Research map / Large language models / Agents, code and math
Learning to Solve Complex Problems via Dataset Decomposition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DECOMP, a teacher-led recursive decomposition of math and code datasets into verified sub-problems with a concept-dependency-graph difficulty score, used to build easy-to-hard curricula for SFT of small LMs.
Problem
Curriculum learning for LMs has yielded mixed results because difficulty ranking within fixed datasets is hard to estimate and datasets lack granularity of atomic skills; the paper asks whether recursively decomposing complex problems into simpler verified sub-problems can enable better curricula fo
Approach
A teacher LLM recursively splits each worked solution into steps, tags each step with a concept, generates grounded sub-problems verified by consistency checks, and scores difficulty via a concept dependency graph depth plus structural branching factor, then trains students easy-to-hard.
Key results
- On MATH-500 with Qwen2.5-1.5B, SFT-Decomp + curriculum reaches 51.6 vs 48.4 for plain SFT on 360 examples and 47.6 for full-dataset SFT.
- On AIME2025 with Qwen3-4B-Base, SFT-Decomp + curriculum reaches 16.7 vs 10.0 base and 3.3 for SFT on AIME2024 data, outperforming Qwen2.5-72B-Instruct (15.0).
- On HumanEval, SFT-Decomp raises DeepSeek-R1-Distill-Qwen-1.5B from 36.01 to 57.90 pass@1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add recursive dataset decomposition to SFT | Qwen2.5-1.5B on MATH · MATH-500 | accuracy: $48.4 \pm 2.2$ -> $50.8 \pm 2.2$ | Helped |
| Add easy-to-hard curriculum to SFT-Decomp (ablation) | Qwen2.5-1.5B on MATH · MATH-500 | accuracy: $50.8 \pm 2.2$ -> $51.6 \pm 2.2$ | Helped |
| Add recursive dataset decomposition to SFT | Qwen3-4B-Base on AIME2024 training data · AIME2025 | accuracy: $3.3 \pm 3.3$ -> $13.3 \pm 6.3$ | Helped |
| Add easy-to-hard curriculum to SFT-Decomp (ablation) | Qwen3-4B-Base on AIME2024 training data · AIME2025 | accuracy: $13.3 \pm 6.3$ -> $16.7 \pm 6.9$ | Helped |
| Add recursive dataset decomposition to SFT | DeepSeek-R1-Distill-Qwen-1.5B on MATH · AIME2025 | accuracy: $23.33 \pm 7.85$ -> $30.00 \pm 8.51 $ | Helped |
| Add recursive dataset decomposition to SFT | Qwen2.5-1.5B-Instruct on Codeforces-CoTs · HumanEval | pass@1: $35.37 \pm 3.74$ -> $42.68 \pm 3.87$ | Helped |
| Add recursive dataset decomposition to SFT | DeepSeek-R1-Distill-Qwen-1.5B on Codeforces-CoTs · HumanEval | pass@1: $36.01 \pm 1.85$ -> $57.90 \pm 0.98$ | Helped |
| Fine-tune on the original AIME2024 dataset with standard SFT | Qwen3-4B-Base · AIME2025 | accuracy: $10.0 \pm 5.6$ -> $3.3 \pm 3.3$ | Hurt |
Surprising findings
- Plain SFT on math datasets often reduced performance relative to the base model, especially on AIME.
- Qwen3-4B-Base trained on decomposed AIME2024 data outperforms the 32B S1 model on AIME2025.
Limitations
- Relies on a strong teacher model (GPT-4o, o4-mini) to decompose and tag problems; hallucinating teachers can yield wrong decompositions.
- Approach may be less effective in domains where reasoning is not easily divisible, such as beyond math and code.
- Small evaluation sets (AIME 30 problems) with large standard deviations (several points) limit statistical confidence.
Experimental setup
Datasets: MATH AIME 2024 AIME 2025 CodeForces-CoTs GSM8K
Benchmarks: MATH-500 AIME 2025 HumanEval
Metrics: accuracy pass@1
Baselines: SFT SFT-Direct Distillation SFT-MetaMATH-Aug SFT-MuggleMath S1-32B Qwen2.5-72B-Instruct GPT-4o Llama3-8B Mistral-7B Gemma2-9B
Builds on: curriculum learning MetaMath MuggleMath s1
Closest papers by meaning
- CurBench: Curriculum Learning Benchmark (ICML 2024 · similarity 0.85)
- K12-KGraph: A Curriculum-Aligned Knowledge Graph for Benchmarking and Training Educational LLMs (NeurIPS 2026 · similarity 0.84)
- Montessori-Instruct: Generate Influential Training Data Tailored for Student Learning (ICLR 2025 · similarity 0.83)
- The Signal is in the Steps: Local Scoring for Reasoning Data Selection (ICML 2026 · similarity 0.83)
- Teaching Models to Teach Themselves: Reasoning at the Edge of Learnability (ICML 2026 · similarity 0.83)
- DataEnvGym: Data Generation Agents in Teacher Environments with Student Feedback (ICLR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.