Research map / Large language models / Alignment and preferences
Perplexity-aware Correction for Robust Alignment with Noisy Preferences
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PerpCorrect, which detects and flips noisy preference labels using a perplexity difference (PPLDiff) computed by a surrogate LLM aligned on a small clean validation set and iteratively refined with reliable training data, then runs standard alignment on the denoised data.
Problem
Alignment of LLMs with preference data containing noisy preferences (chosen and rejected labels flipped), which degrades DPO/PPO alignment performance.
Approach
Detect noisy preferences from the data side: the perplexity difference between chosen and rejected responses under an aligned surrogate LLM separates clean from noisy pairs; a two-Gaussian fit via Levenberg-Marquardt gives a threshold, and flagged pairs have their labels flipped before alignment. Iterative surrogate alignment with confidently clean and confidently noisy training samples sharpens t
Key results
- At 40% noisy preferences on Golden HH with Llama2-7B, PerpCorrect-DPO raises reward accuracy from 53.15% (vanilla DPO) to 94.92%.
- With phi-2 on Golden HH at 40% noise, PerpCorrect-DPO reaches 96.39% versus 54.98% for vanilla DPO and 84.55% for rDPO.
- On OASST1 with phi-2 at 40% noise, PerpCorrect-DPO reaches 68.49% versus 52.42% for vanilla DPO; combined with cDPO it lifts results at high noise levels.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply PerpCorrect correction before DPO alignment | DPO with Llama2-7B · Golden HH, 40% noisy preferences | reward accuracy: 53.15% -> 94.92% | Helped |
| Apply PerpCorrect correction before DPO alignment | DPO with phi-2 · Golden HH, 40% noisy preferences | reward accuracy: 54.98% -> 96.39% | Helped |
| Apply PerpCorrect correction before DPO alignment | DPO with phi-2 · OASST1, 40% noisy preferences | reward accuracy: 52.42% -> 68.49% | Helped |
| Apply PerpCorrect correction before PPO alignment | PPO with Llama2-7B · Golden HH, 10% noisy preferences | reward accuracy: 96.64% -> 96.38% | Hurt |
| Apply PerpCorrect correction before PPO alignment | PPO with Llama2-7B · Golden HH, 40% noisy preferences | reward accuracy: 90.62% -> 93.17% | Helped |
| Combine PerpCorrect with rDPO | rDPO with Llama2-7B · Golden HH, 10% noisy preferences | reward accuracy: 96.65% -> 95.99% | Hurt |
| Combine PerpCorrect with rDPO | rDPO with Llama2-7B · Golden HH, 40% noisy preferences | reward accuracy: 90.45% -> 95.73% | Helped |
| Increase clean validation samples from 10 to 50 (ablation) | PerpCorrect-DPO with Llama2-7B · Golden HH, 40% noisy preferences | reward accuracy: 81.40% -> 95.43% | Helped |
| Increase clean validation samples from 50 to 200 (ablation) | PerpCorrect-DPO with Llama2-7B · Golden HH, 40% noisy preferences | reward accuracy: 95.43% -> 96.04% | Helped |
| Increase clean validation samples from 30 to 40 (ablation) | PerpCorrect-DPO with Llama2-7B · Golden HH, 40% noisy preferences | reward accuracy: 94.21% -> 94.21% | No clear effect |
Surprising findings
- An unaligned surrogate LLM's PPLDiff cannot separate clean from noisy preferences; alignment of the surrogate is required for the separation.
- On the OASST1 dataset, the other robust methods (cDPO, rDPO) often fail to beat vanilla DPO.
Limitations
- Iterative PPLDiff computation and surrogate-model training roughly double the cost of standard alignment.
- Requires a clean validation set (about 50 samples); performance is much lower with 10 samples.
- Evaluated only on two datasets, two models, and random label-flip noise.
Experimental setup
Datasets: OASST1 Golden HH Alpaca
Benchmarks: Golden HH test split OASST1 test split
Metrics: reward accuracy (win rate against selected preferences)
Baselines: vanilla DPO cDPO rDPO vanilla PPO cPPO rPPO SLiC IPO
Builds on: DPO cDPO rDPO RLHF QLoRA
Closest papers by meaning
- ComPO: Preference Alignment via Comparison Oracles (NeurIPS 2025 · similarity 0.90)
- Aligner: Efficient Alignment by Learning to Correct (NeurIPS 2024 · similarity 0.89)
- Clean First, Align Later: Benchmarking Preference Data Cleaning for Reliable LLM Alignment (NeurIPS 2025 · similarity 0.89)
- TODO: Enhancing LLM Alignment with Ternary Preferences (ICLR 2025 · similarity 0.89)
- SimPER: A Minimalist Approach to Preference Alignment without Hyperparameters (ICLR 2025 · similarity 0.89)
- Leveraging robust optimization for llm alignment under distribution shifts (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.