Research map / Large language models / Alignment and preferences
Iterative Label Refinement Matters More than Preference Optimization under Weak Supervision
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows DPO fails to improve SFT under unreliable demonstrations and comparisons, and proposes iterative label refinement (ILR), which uses comparison feedback to replace SFT demonstrations and retrains via SFT, outperforming SFT+DPO on math, SQL, and safe instruction-following tasks.
Problem
Post-training (SFT followed by RLHF/DPO) may become unreliable when tasks are hard for humans to supervise; it is unclear whether SFT and preference optimization remain effective under unreliable demonstrations and comparisons.
Approach
Instead of using comparison feedback to continually train the model via DPO, ILR uses the feedback to decide whether to replace unreliable SFT demonstrations with model-generated proposals (cross-labeled by two models trained on disjoint halves), then retrains the model from scratch via SFT on the refined data each round.
Key results
- In LM-simulated settings, SFT+ILR consistently outperforms SFT+DPO across four unreliable-supervision settings on GSM8K, BIRD, and SaferPaca.
- DPO with unreliable comparisons shows little to no gain over SFT and often declines, while DPO with oracle comparisons does improve over the SFT starting point.
- In the time-constrained human study on Alpaca, ILR shows fast and consistent improvement over two rounds, while DPO with small beta plateaus; humans preferred model outputs over their own demonstrations for over 20% of questions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use ILR instead of DPO | GSM8K (2B → 7B) · GSM8K (2B → 7B), four rounds | accuracy: DPO: 0.3591, 0.3520, 0.3462, 0.3412; ILR: 0.3647, 0.3738, 0.3851, 0.3788 | Helped |
| add disjoint-data models to generate preference samples (sDPO) (ablation) | DPO · GSM8K (2B → 7B), four rounds | accuracy: DPO: 0.3591, 0.3520, 0.3462, 0.3412; sDPO: 0.3624, 0.3561, 0.3457, 0.3343 | Mixed |
| compare model-generated responses with original unreliable labels (wsDPO) (ablation) | DPO · GSM8K (2B → 7B), four rounds | accuracy: DPO: 0.3591, 0.3520, 0.3462, 0.3412; wsDPO: 0.3626, 0.3624, 0.3538, 0.3482 | Helped |
| replace standard DPO loss with IPO (ablation) | DPO · GSM8K (2B → 7B), four rounds | accuracy: DPO: 0.3591, 0.3520, 0.3462, 0.3412; IPO: 0.3604, 0.3571, 0.3503, 0.3460 | Helped |
| replace standard DPO loss with cDPO (ablation) | DPO · GSM8K (2B → 7B), four rounds | accuracy: DPO: 0.3591, 0.3520, 0.3462, 0.3412; cDPO: 0.3589, 0.3536, 0.3563, 0.3530 | Mixed |
| replace standard DPO loss with rDPO (ablation) | DPO · GSM8K (2B → 7B), four rounds | accuracy: DPO: 0.3591, 0.3520, 0.3462, 0.3412; rDPO: 0.3568, 0.3487, 0.3386, 0.3391 | Hurt |
| use the 15% most confident feedback instead of all feedback (ablation) | DPO · GSM8K (2B → 7B), one round | accuracy: 0.32±0.0043 -> 0.36±0.0032 | Helped |
| remove comparison feedback and replace labels directly with model proposals (ablation) | ILR · GSM8K, BIRD, and SaferPaca | performance | Hurt |
| set refinement rate α to 0.05 (ablation) | ILR · GSM8K | label accuracy | Hurt |
| use larger refinement rates α (ablation) | ILR · GSM8K | performance stability | Hurt |
| use an oracle evaluator for DPO comparisons (ablation) | DPO initialized from unreliable-data SFT · GSM8K | performance | Helped |
Surprising findings
- DPO fails to improve over SFT under unreliable demonstrations and comparisons, while it works with oracle comparisons, suggesting its effectiveness heavily depends on supervision quality.
- Naively replacing SFT labels with model proposals without feedback degrades performance, showing that even unreliable comparison feedback is necessary for refinement.
Limitations
- Focuses on DPO and does not test PPO or reward-model-based RLHF pipelines.
- Time-constrained human study may not faithfully simulate realistic human errors; tasks are not beyond human capability.
- Benefit of ILR diminishes over rounds as humans select fewer AI proposals.
Experimental setup
Datasets: GSM8K BIRD SaferPaca Alpaca AlpacaEval I-MaliciousInstructions I-CoNA
Benchmarks: GSM8K BIRD SaferPaca AlpacaEval
Metrics: exact match accuracy execution accuracy win rate vs reference (GPT-4o judge)
Baselines: SFT SFT+DPO sDPO wsDPO IPO cDPO rDPO
Builds on: direct preference optimization weak-to-strong generalization co-teaching LoRA
Closest papers by meaning
- A Critical Evaluation of AI Feedback for Aligning Large Language Models (NeurIPS 2024 · similarity 0.88)
- Vanishing Gradients in Reinforcement Finetuning of Language Models (ICLR 2024 · similarity 0.87)
- Let's Verify Step by Step (ICLR 2024 · similarity 0.87)
- RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback (ICML 2024 · similarity 0.87)
- SELF-EVOLVED REWARD LEARNING FOR LLMS (ICLR 2025 · similarity 0.87)
- Explainable Reinforcement Learning from Human Feedback to Improve Alignment (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.