Research map / Large language models / Reasoning and chain of thought
Mind the Gap: Examining the Self-Improvement Capabilities of Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Formalizes LLM self-improvement as a generation-verification gap (GV-Gap) and empirically studies its scaling, cross-verification, unimprovable tasks, iterative self-improvement, and verifier ensembles across base models from several families.
Problem
Self-improvement via self-verification and reweighted distillation is widely used, but its fundamental capabilities and confounders are poorly understood; existing studies focus on single families or verification mechanisms.
Approach
Defines the generation-verification gap (GV-Gap) as the expected-utility improvement from reweighting a model's generations by its own verification score, and separates verification from model update to measure self-improvement in a modular, controlled way; finds relative gap scales roughly linearly with log pre-training FLOPs for some verifiers.
Key results
- Relative GV-Gap for CoT-Score increases monotonically with pre-training FLOPs across Qwen-1.5 models; small models (0.5B-7B Llama-2, Qwen-1.5 0.5B) show non-positive gap for nearly all verifiers.
- On Natural Questions, gaps for Qwen-2 models are all within about ±1%, indicating the task is unimprovable under this framework.
- On 4x4 Sudoku only Qwen-2 72B shows a sizeable gap (16.99 points, relative gap 20.81%), with Llama-3.1 70B showing a positive gap as well.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MC top 0.8 self-verification | Qwen-1.5 7B · GSM8K | gap: +7.42 | Helped |
| Use CoT-Binary self-verification | Qwen-1.5 7B · GSM8K | gap: +4.07 | Helped |
| Use self-verification on factual questions | Qwen-2 models · Natural Question | gap: -0.06 to 0.79 | No clear effect |
| Use self-verification with CoT reasoning | Qwen-2 72B · Sudoku | gap: +16.99 | Helped |
| Use self-verification with CoT reasoning | Qwen-2 0.5B · Sudoku | gap: -0.09 | Hurt |
| Combine non-trivial verification methods using AND filtering | Qwen-2 model family · GSM8K | relative gap: 4.66 -> 5.24 | Helped |
| Lower sampling temperature from 0.7 to 0.5 (ablation) | Qwen-1.5 model family · GSM8K | gap | Hurt |
| Raise sampling temperature from 0.7 to 1 (ablation) | Qwen-1.5 model family · GSM8K | relative gap scaling | Mixed |
| Use instruct models instead of base models (ablation) | Qwen-1.5-Chat and Qwen-2.5-Instruct · GSM8K and MATH | relative gap scaling | Hurt |
| Keep the verifier fixed to gold labels during iterative training (ablation) | Qwen-1.5 7B · GSM8K | effective generation diversity | Helped |
| Keep the verifier fixed to Qwen-1.5 72B (ablation) | Qwen-1.5 7B · GSM8K | effective generation diversity | No clear effect |
| Iterate self-improvement with CoT-Binary verification | Qwen-1.5 models · GSM8K | gap | Mixed |
Surprising findings
- Only the largest models self-improve on Sudoku, even though verification is simpler than generation, while most models self-improve on GSM8K.
- Gaps of verification methods do not positively correlate with generation accuracy.
Limitations
- Only one verification sample per generation due to computational constraints.
- Scaling trend is not consistent across all verifiers or model families; the absolute gap shows no clear scaling.
- Instruct models show noisier or absent scaling, so conclusions are restricted to base models.
Experimental setup
Datasets: GSM8K MATH Natural Questions 4x4 Sudoku puzzles
Benchmarks: GSM8K MATH Natural Questions Sudoku
Metrics: generation-verification gap relative gap accuracy pass@k
Baselines: Multiple Choice verification CoT-Score verification CoT-Binary verification Tournament verification
Builds on: rejection sampling fine-tuning KL-regularized RL / RLHF chain-of-thought prompting LLM-as-a-judge
Closest papers by meaning
- Self-Improvement in Language Models: The Sharpening Mechanism (ICLR 2025 · similarity 0.95)
- Latent Principle Discovery for Language Model Self-Improvement (NeurIPS 2025 · similarity 0.91)
- Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizing (NeurIPS 2024 · similarity 0.91)
- Learning to Self-Verify Makes Language Models Better Reasoners (ICML 2026 · similarity 0.91)
- Large Language Models Cannot Self-Correct Reasoning Yet (ICLR 2024 · similarity 0.90)
- Importance Weighting Can Help Large Language Models Self-Improve (AAAI 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.