Research map / Large language models / Reasoning and chain of thought
Teaching Models to Teach Themselves: Reasoning at the Edge of Learnability
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SOAR, an asymmetric teacher-student self-play framework where a teacher LLM is meta-RL-trained to generate stepping-stone question-answer pairs, rewarded by the student's improvement on a hard dataset without the teacher seeing the hard problems.
Problem
RLVR stalls on datasets where the model's initial success rate is near zero, giving no training signal; the paper asks whether a pretrained LLM can generate its own curriculum of easier problems that unlock learning on such hard problems.
Approach
Nested meta-RL: a teacher (RLOO) generates question-answer pairs, a student is trained on them with RLOO for a few steps, and the teacher is rewarded with the student's accuracy gain on a grounded hard training set; a promotion mechanism updates the student baseline, and the teacher never sees the hard problems.
Key results
- PQ-MATH yields +9.3% pass@32 over Hard-Only on fail@128 MATH test; PS yields +8.5% pass@32 on MATH.
- On hard subsets, self-generated problems give roughly 4x pass@1 and 2x pass@32 on MATH, and 2x pass@1 and 1.5x pass@32 on HARP.
- Intrinsic learnability teacher collapses in diversity (Vendi Score 10.82 vs 34.91 for base teacher); only 32.8% of PQ problems have fully correct solutions while 63% are well-posed.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| grounded meta-RL teacher (SOAR) generated promotion questions added to fail@128 training | Llama-3.2-3B-Instruct student on fail@128-MATH · fail@128-MATH test | pass@32: +9.3% (PQ) vs Hard-Only | Helped |
| grounded meta-RL teacher (SOAR) generated promotion questions added to fail@128 training | Llama-3.2-3B-Instruct student on fail@128-HARP · fail@128-HARP test | pass@32: +4.2% (PQ) vs Hard-Only | Helped |
| 4x group size for direct RL on hard problems (Hard-Only with group size 128) (ablation) | Hard-Only baseline on fail@128-MATH · fail@128-MATH test | pass@32: +2.8% vs much larger gains from PQ | Hurt |
| learnability (intrinsic) reward for teacher instead of grounded student-progress reward (ablation) | Teacher model generating training questions · Synthetic dataset semantic diversity (Vendi Score) | Vendi Score: 10.82 vs 34.66 (Grounded-T HARP) | Hurt |
| grounded student-progress reward instead of intrinsic learnability reward (ablation) | Teacher model training · Student test performance across teacher seeds | pass@32 student trajectories | Helped |
| grounded student-progress reward instead of intrinsic learnability reward (ablation) | Teacher model training · fail@128-MATH/HARP/OlympiadBench | pass@32 | Helped |
| SeRL self-play curriculum baseline instead of grounded teacher | Llama-3.2-3B-Instruct on fail@128 sets · fail@128-MATH/HARP | pass@32 | Hurt |
| training on synthetic questions from PQ-MATH, PQ-HARP, Intrinsic-T | Student transferred to OlympiadBench (OOD) · OlympiadBench | pass@k | Helped |
| well-posed questions with incorrect answers included in training versus only correct-answer questions (ablation) | HARP-PQ training data · HARP fail@128 | performance | Helped |
| Grounded teacher trained with meta-RL, direct inference on fail@128 test | Trained teacher policy as solver · fail@128 test | pass@k | No clear effect |
Surprising findings
- Only 32.8% of self-generated questions contain fully correct solutions, yet they still provide useful learning signal.
- Teacher questions that are useful do not require the teacher to be able to solve the hard problems; direct inference with the trained teacher does not improve fail@128 accuracy.
Limitations
- High computational cost of nested bilevel RL loops, requiring parallel student training for stability.
- Only evaluated on 3B and 8B models; scaling beyond this is untested.
- Relies on fail@128 hard subsets with ground-truth answers for grounding reward; generated questions often have incorrect answers.
Experimental setup
Datasets: MATH HARP OlympiadBench
Benchmarks: MATH fail@128 test HARP fail@128 test OlympiadBench fail@128 test
Metrics: pass@k (k=1,4,8,16,32) Vendi Score
Baselines: Hard-Only Intrinsic-T SeRL Base-T
Builds on: RLOO asymmetric self-play dataset distillation learnability reward
Closest papers by meaning
- Improving Rationality in the Reasoning Process of Language Models through Self-playing Game (ICML 2025 · similarity 0.89)
- SwS: Self-aware Weakness-driven Problem Synthesis in Reinforcement Learning for LLM Reasoning (NeurIPS 2025 · similarity 0.89)
- R-Diverse: Mitigating Diversity Illusion in Self-Play LLM Training (ICML 2026 · similarity 0.89)
- h1: Bootstrapping LLMs to Reason over Longer Horizons via Reinforcement Learning (ICML 2026 · similarity 0.88)
- B-STaR: Monitoring and Balancing Exploration and Exploitation in Self-Taught Reasoners (ICLR 2025 · similarity 0.88)
- RLKD: Distilling LLMs’ Reasoning via Reinforcement Learning (AAAI 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.