Research map / Large language models / Alignment and preferences

Clean First, Align Later: Benchmarking Preference Data Cleaning for Reliable LLM Alignment

NeurIPS 2025 · Large language models › Alignment and preferences · approach: Steering and correcting LLM behavior · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Benchmarks 13 preference data cleaning methods (LLM-as-judge, reward-model based, heuristic) for LLM alignment across four preference datasets, base models, and preference optimization algorithms, finding that multi-RM voting with removal works best and generalizes well.

Problem

Human preference data for LLM alignment is noisy and inconsistent, but there is no systematic, standardized comparison of automated cleaning methods or of how well they generalize across datasets, models, and optimizers.

Approach

A unified benchmarking protocol (PrefCleanBench) that groups 13 cleaning approaches under identification criteria (LLM judge, reward models, heuristics) with remove or flip treatments, and evaluates them by win-tie rate and gold reward of models trained on cleaned data across datasets, base models, and DPO-family algorithms.

Key results

• VoteMaj-R (remove data flagged by majority of six reward models) achieves top average reward on Llama3-8B DPO across datasets, e.g., 7.287 on Anthropic-HH vs 6.001 vanilla and 8.478 on PKU-SafeRLHF vs 6.318.
• Removing unreliable data generally beats flipping labels; e.g., VoteMaj-R outperforms VoteMaj-F on Anthropic-HH win-tie 0.705 vs 0.695 and reward 7.287 vs 7.010, with flipping degrading results on HelpSteer2.
• VoteMaj-R keeps win-tie above 0.5 across Llama3.2-1B, Qwen2.5-7B, Mistral-7B and phi-2, whereas Tag-Cmp drops below 0.5 in some settings (e.g., phi-2 on Anthropic-HH 0.395).

Measured effects reported in the paper

Surprising findings

• Flipping labels often degrades performance relative to removal, since unreliability often stems from low-quality prompts or harmful responses rather than mislabeling.
• LLM-as-a-judge cleaning underperforms multi-RM voting largely because it must pick a winner when both responses are similarly harmful, keeping such data about half the time.

Limitations

• No ground truth for preference correctness, so cleaning is evaluated only via downstream alignment, not recall or false positive rate.
• Evaluation relies on GPT-4o judge and reward models that may be biased; win-tie judged on 200 sampled test points.
• Hyperparameter choice for removal ratio (RwGap, IFD) is selected as optimal on test results, which may overstate those baselines.

Experimental setup

Datasets: Anthropic-HH UltraFeedback PKU-SafeRLHF HelpSteer2

Benchmarks: Anthropic-HH UltraFeedback PKU-SafeRLHF HelpSteer2

Metrics: win-tie rate average gold reward

Baselines: LLM-Judge-R LLM-Judge-F RwGap-R RwGap-F VoteAll-R VoteMaj-R Tag-Cmp IFD-Gap-R

Builds on: DPO InsTag instruction following difficulty reward gap filtering RewardBench

Closest papers by meaning

• Reward-Augmented Data Enhances Direct Preference Alignment of LLMs (ICML 2025 · similarity 0.92)
• Larger or Smaller Reward Margins to Select Preferences for LLM Alignment? (ICML 2025 · similarity 0.92)
• On Evaluating LLM Alignment by Evaluating LLMs as Judges (NeurIPS 2025 · similarity 0.91)
• ComPO: Preference Alignment via Comparison Oracles (NeurIPS 2025 · similarity 0.91)
• Principled Data Selection for Alignment: The Hidden Risks of Difficult Examples (ICML 2025 · similarity 0.91)
• Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Models (ICLR 2024 · similarity 0.91)