Research map / Large language models / Reasoning and chain of thought
Progress or Regress? Self-Improvement Reversal in Post-training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies iterative post-training (iterative SFT, DPO, SFT-DPO) for LLM self-improvement on math, commonsense and code tasks and proposes metrics (improvement set pass@N, solution diversity, OOD generalization) revealing a "self-improvement reversal" where pass@1 gains coincide with lost diversity and OOD capability.
Problem
Whether iterative self-improvement via post-training truly improves problem-solving on harder problems, or whether pass@1 gains hide regressions in output diversity and out-of-distribution generalization.
Approach
Decouples factors of iterative self-improvement (base model, task, iterations, post-training method) and proposes an evaluation framework beyond pass@1: pass@N on the improvement set, solution diversity metrics, and OOD group-disparity analysis, showing pass@1 gains come from answer selection rather than new capability.
Key results
- Iterative post-training raises pass@1 across benchmarks, but gains plateau or decline after 4-5 iterations; e.g., LLaMA2-7B Iterative SFT gains +12.31 on GSM8K, and LLaMA3-8B reaches 69.06 vs 53.91 for LLaMA2-7B optimal accuracy.
- Pass@N of the first-iteration model M1 on improvement sets approaches near-perfect as N grows, suggesting gains stem from answer selection rather than new problem-solving ability.
- Solution diversity (distinct n-grams, Sentence-BERT, distinct equations) consistently decreases over iterations for all three methods, and Iterative SFT and SFT-DPO harm MATH algebra OOD accuracy while DPO widens group disparity.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Iterative post-training iterations beyond 4-5 | iterative SFT/DPO/SFT-DPO self-improvement · CSQA, GSM8K, MATH, MBPP | pass@1 accuracy | Mixed |
| Adding iterative post-training (self-improvement) over base model | LLaMA2-7B, Mistral-7B, LLaMA3-8B · GSM8K | pass@1 accuracy: +12.31 (LLaMA2-7B Iterative SFT) | Helped |
| Iterative DPO vs Iterative SFT in OOD transfer | Mistral-7B trained on GSM8K, tested on MATH algebra · MATH algebra (OOD) | whole accuracy and group disparity | Mixed |
| Iterative SFT and Iterative SFT-DPO post-training on GSM8K | Mistral-7B · MATH algebra (OOD) | OOD generalization accuracy | Hurt |
| Iterative self-improvement (SFT, DPO, SFT-DPO) | Mistral-7B · all datasets | output diversity (distinct n-grams, SBERT, distinct equations) | Hurt |
| Iterative DPO vs Iterative SFT | Mistral-7B · GSM8K | semantic diversity (SBERT) | Mixed |
| Choosing Iterative DPO / SFT-DPO over Iterative SFT when correct answer coverage of M1 is high (>0.5) | post-training method selection · CSQA, GSM8K, MATH, MBPP | optimal pass@1 of M*_t | Helped |
| Iterative SFT when correct answer coverage of M1 is low (<=0.5) | post-training method selection · CSQA, GSM8K, MATH, MBPP | optimal pass@1 of M*_t | Helped |
| Iterative SFT-DPO scaling to LLaMA-2-70B | LLaMA-2-70B on GSM8K · GSM8K | pass@1 accuracy | Mixed |
Surprising findings
- Pass@1 improvements from iterative post-training coincide with a reduction in OOD generalization on harder MATH levels, with DPO's apparent OOD gain driven by easier problems.
- Gains in pass@1 do not reflect acquisition of new problem-solving abilities; M1 with pass@N already covers the improvement problems.
Limitations
- Experiments limited to 7B-8B models (plus one 70B scaling run) and four tasks due to compute.
- Proposed metrics may not capture all dimensions of model performance, especially in real-world settings.
- The correct answer coverage hypothesis is preliminary and not theoretically validated.
Experimental setup
Datasets: CommonsenseQA GSM8K MATH MBPP
Benchmarks: CommonsenseQA validation GSM8K MATH MBPP
Metrics: pass@1 accuracy pass@N on improvement set distinct n-grams Sentence-BERT cosine diversity distinct equations group disparity on MATH algebra
Baselines: Iterative SFT Iterative DPO Iterative SFT-DPO
Builds on: STaR DPO self-training
Closest papers by meaning
- Mind the Gap: Examining the Self-Improvement Capabilities of Large Language Models (ICLR 2025 · similarity 0.90)
- A Task-centric Theory for Iterative Self-Improvement with Easy-to-Hard Curricula (ICML 2026 · similarity 0.90)
- Self-Improvement in Language Models: The Sharpening Mechanism (ICLR 2025 · similarity 0.89)
- Enabling Lanuguage Models to Implicitly Learn Self-Improvement (ICLR 2024 · similarity 0.88)
- B-STaR: Monitoring and Balancing Exploration and Exploitation in Self-Taught Reasoners (ICLR 2025 · similarity 0.88)
- Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizing (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.