Research map / Large language models / Instruction and fine-tuning
Importance Weighting Can Help Large Language Models Self-Improve
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes IWSI, an LLM self-improvement method that filters self-generated reasoning samples by self-consistency plus a DS weight (a naive importance-weighting estimate of distribution shift from a tiny valid set), outperforming LMSI and matching a reward-model filter.
Problem
LLM self-improvement on self-generated data suffers from variable data quality; existing filters only check answer correctness/consistency and ignore distribution shift between self-generated and real data.
Approach
Estimate each self-generated sample's distribution shift extent (DSE) with a naive importance-weight approximation (DS weight) computed from the mean SFT loss on a tiny human-written valid set, then keep the lowest-DS-weight samples among majority-consistent ones before fine-tuning.
Key results
- IWSI averages 40.4% accuracy across six datasets versus 34.0% for LMSI and 22.1% for the base Llama3-8B, matching RM-filter's 40.4%.
- IWSI reaches 37.6 on gsm8k and 62.7 on SVAMP, and is the only self-improvement method that consistently outperforms LMSI.
- IWSI-w (weighted loss variant) performs worse than IWSI, averaging 34.2% on the same setup.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add DS-weight filtering to majority-vote self-consistency filtering | Llama3-8B self-improvement (IWSI vs LMSI) · GSM8K | accuracy: 27.9 -> 37.6 | Helped |
| Add DS-weight filtering to majority-vote self-consistency filtering | Llama3-8B self-improvement (IWSI vs LMSI) · SVAMP | accuracy: 45.0 -> 62.7 | Helped |
| Add DS-weight filtering to majority-vote self-consistency filtering | Llama3-8B self-improvement (IWSI vs LMSI) · ANLI-A1 | accuracy: 25.2 -> 27.2 | Helped |
| Add DS-weight filtering to majority-vote self-consistency filtering | Llama3-8B self-improvement (IWSI vs LMSI) · ANLI-A2 | accuracy: 22.6 -> 23.4 | Helped |
| Add DS-weight filtering to majority-vote self-consistency filtering | Llama3-8B self-improvement (IWSI vs LMSI) · OpenBookQA | accuracy: 31.6 -> 37.0 | Helped |
| Add DS-weight filtering to majority-vote self-consistency filtering | Llama3-8B self-improvement (IWSI vs LMSI) · StrategyQA | accuracy: 51.4 -> 54.6 | Helped |
| Use DS weight in weighted loss instead of ranking and filtering samples (ablation) | IWSI · SVAMP | accuracy: 62.7 -> 43.3 | Hurt |
| Use DS weight in weighted loss instead of ranking and filtering samples (ablation) | IWSI · OpenBookQA | accuracy: 37.0 -> 31.8 | Hurt |
| Vary filtering percentage k away from the appropriate range (ablation) | IWSI · GSM8K, StrategyQA, and ANLI-A1 | accuracy | Mixed |
Surprising findings
- Correct self-generated samples with high distribution shift extent can degrade self-improvement, so filtering by correctness alone is insufficient.
- Using DS weight as a weight in the training loss (IWSI-w) underperforms using it only to rank and filter samples.
Limitations
- IWSI cannot stop model collapse in repeated self-consuming iterations.
- The naive DS weight estimator ignores semantic similarity among training samples, which hurts the weighted-loss variant.
- Valid set is human-written and small; results vary across datasets and the optimal filtering threshold k varies by task.
Experimental setup
Datasets: gsm8k SVAMP ANLI-A1 ANLI-A2 OpenBookQA StrategyQA
Benchmarks: gsm8k SVAMP ANLI-A1 ANLI-A2 OpenBookQA StrategyQA
Metrics: accuracy
Baselines: LMSI Entropy-filter Self-filter IWSI-w RM-filter
Builds on: LMSI DIW Self-Consistency MoT Self-Alignment
Closest papers by meaning
- Mind the Gap: Examining the Self-Improvement Capabilities of Large Language Models (ICLR 2025 · similarity 0.90)
- Exploring Criteria of Loss Reweighting to Enhance LLM Unlearning (ICML 2025 · similarity 0.90)
- Latent Principle Discovery for Language Model Self-Improvement (NeurIPS 2025 · similarity 0.90)
- Dynamic Loss-Based Sample Reweighting for Improved Large Language Model Pretraining (ICLR 2025 · similarity 0.90)
- Self-Improvement in Language Models: The Sharpening Mechanism (ICLR 2025 · similarity 0.89)
- LLM Data Selection and Utilization via Dynamic Bi-level Optimization (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.