Research map / Large language models / Language, safety and interpretability
Self-Improvement in Language Models: The Sharpening Mechanism
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Formalizes LLM self-improvement as "sharpening" toward high-likelihood responses, proves sample-complexity bounds for SFT- and RLHF-based sharpening (with XPO exploration removing the coverage dependence), and validates with best-of-N and SFT experiments on several models.
Problem
Self-improvement without external feedback seems to create no new information, so it is unclear why it should improve capabilities; the paper asks when and how a model can be sharpened toward its own high-likelihood (self-rewarded) responses and what the fundamental limits are.
Approach
Frame self-improvement as sharpening: use the model's own sequence log-likelihood as a self-reward and amortize best-of-N-style maximum-likelihood inference into post-training, analyzed in a sample-and-evaluate statistical framework where SFT-based sharpening is minimax optimal under coverage and exploratory RLHF (XPO) can remove the coverage dependence.
Key results
- Best-of-N sharpening with N=50 improves over greedy decoding on at least one model for every dataset; SFT-Sharpening on Phi3.5-Mini yields +19.24% accuracy lift over greedy on MATH.
- SFT-Sharpening gives +12.46% lift on ProntoQA and +1.82% on GSM8k for Phi3.5-Mini, and +8.88% on MATH for Mistral-7B-Instruct-v0.3.
- Theory: SFT-Sharpening achieves minimax-optimal sample complexity under coverage, while XPO-based RLHF-Sharpening removes dependence on the coverage coefficient using the SEC, with an exponential improvement for linear softmax models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train on best-of-50 responses selected by sequence log-likelihood | Phi3.5-Mini · MATH | accuracy: 19.24 ± 2.41% lift over greedy | Helped |
| Train on best-of-50 responses selected by sequence log-likelihood | Phi3.5-Mini · MATH | likelihood: 48.33 ± 0.17 | Helped |
| Train on best-of-50 responses selected by sequence log-likelihood | Phi3.5-Mini · GSM8K | accuracy: 1.82 ± 0.64% lift over greedy | Helped |
| Train on best-of-50 responses selected by sequence log-likelihood | Phi3.5-Mini · GSM8K | likelihood: 1.49 ± 0.55 | Helped |
| Train on best-of-50 responses selected by sequence log-likelihood | Phi3.5-Mini · ProntoQA | accuracy: 12.46 ± 1.08% lift over greedy | Helped |
| Train on best-of-50 responses selected by sequence log-likelihood | Phi3.5-Mini · ProntoQA | likelihood: 5.64 ± 0.01 | Helped |
| Train on best-of-50 responses selected by sequence log-likelihood | Mistral-7B-Instruct-v0.3 · MATH | accuracy: 8.88 ± 5.55% lift over greedy | Helped |
| Train on best-of-50 responses selected by sequence log-likelihood | Mistral-7B-Instruct-v0.3 · MATH | likelihood: 5.71 ± 3.00 | Helped |
| Increase inference-time best-of-N using sequence log-likelihood | language models · reasoning tasks | accuracy | Mixed |
| Use length-normalized log-likelihood instead of sequence log-likelihood for best-of-N selection | language models · reasoning tasks | accuracy | Helped |
| Use majority voting instead of sequence log-likelihood for best-of-N selection | language models · reasoning tasks | accuracy | Helped |
| Increase the training-set best-of-N from 10 to 25 or 50 (ablation) | Phi3.5-Mini SFT-Sharpening · MATH | accuracy | Helped |
Surprising findings
- Best-of-N on likelihood can degrade performance as N grows because it favors short responses that forgo chain-of-thought.
- Log-probability is a noisy proxy for accuracy on GSM8k during SFT-Sharpening training.
Limitations
- Maximum-likelihood self-reward is a stylized choice; more sophisticated self-rewards (model-as-judge) are not analyzed theoretically.
- Theoretical RLHF guarantees require realizability, concentrability, and margin assumptions; the coverage-free result depends on bounded SEC and bounded log-probabilities.
- SFT amortization experiments are limited to a small subset of model-task pairs and only a few training runs.
Experimental setup
Datasets: GSM8k MATH ProntoQA MMLU GameOf24
Benchmarks: GSM8k MATH ProntoQA MMLU GameOf24
Metrics: percent accuracy lift over greedy decoding sequence log-likelihood lift
Baselines: greedy decoding temperature 1.0 sampling coverage (pass@N skyline)
Builds on: DPO XPO best-of-N sampling RLHF
Closest papers by meaning
- Mind the Gap: Examining the Self-Improvement Capabilities of Large Language Models (ICLR 2025 · similarity 0.95)
- Enabling Lanuguage Models to Implicitly Learn Self-Improvement (ICLR 2024 · similarity 0.91)
- Latent Principle Discovery for Language Model Self-Improvement (NeurIPS 2025 · similarity 0.91)
- Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizing (NeurIPS 2024 · similarity 0.90)
- Training Language Models to Self-Correct via Reinforcement Learning (ICLR 2025 · similarity 0.90)
- A Task-centric Theory for Iterative Self-Improvement with Easy-to-Hard Curricula (ICML 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.