Research map / Large language models / Alignment and preferences

Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimization

ICLR 2025 · Large language models › Alignment and preferences · approach: Robust and calibrated DPO · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Analyzes DPO through distributionally robust optimization and proposes Dr. DPO, which adds a robustness temperature β' that reweights preference pairs via a log-sum-exp loss to resist pairwise label-flip noise.

Problem

DPO is sensitive to noisy preference data, including low-quality responses (pointwise noise) and flipped preference pairs (pairwise noise); its robustness to these noise types is unclear.

Approach

Views DPO as implicitly performing pointwise DRO with KL divergence (β acts as a noise-reflecting radius parameter), then robustifies against pairwise noise by optimizing a worst-case reweighting of preference pairs, yielding loss -β' log E[exp(h_DPO/β')] that adds only one line of code.

Key results

• On Anthropic HH with 40% flipped pairs, Dr. DPO reaches 58.83% preference accuracy vs 55.23% for DPO, and 61.65% win rate vs 49.00% for DPO.
• In the noise-free setting, Dr. DPO raises HH win rate from 54.36% (DPO) to 56.67% and preference accuracy from 63.63% to 66.22%.
• Dr. DPO with beta'=1.0 is presented as a default setting that needs no per-noise tuning, with optimal beta' decreasing as noise increases.

Measured effects reported in the paper

Surprising findings

• Lower DPO beta (0.01) outperforms higher beta (0.1) on IMDB when trained on 100% low-quality SFT data, supporting the claim that beta reflects SFT noise.
• Dr. DPO improves over DPO even on the 0% flipped HH dataset, which the authors attribute to inherent label noise in the original data.

Limitations

• beta' may be sensitive to data and task, requiring a search in some applications
• Evaluation of pairwise noise uses random label flipping, which may not capture complex real-world noise
• Win-rate evaluation relies on GPT-4 as judge

Experimental setup

Datasets: IMDB Anthropic HH

Benchmarks: Anthropic HH IMDB MT-Bench

Metrics: preference accuracy win rate expected reward frontier KL divergence

Baselines: DPO cDPO IPO rDPO

Builds on: DPO RLHF cDPO IPO rDPO

Closest papers by meaning

• Provably Robust DPO: Aligning Language Models with Noisy Feedback (ICML 2024 · similarity 0.95)
• ROPO: Robust Preference Optimization for Large Language Models (ICML 2025 · similarity 0.94)
• Robust LLM Alignment via Distributionally Robust Direct Preference Optimization (NeurIPS 2025 · similarity 0.94)
• Less is More: Improving LLM Alignment via Preference Data Selection (NeurIPS 2025 · similarity 0.94)
• Leveraging robust optimization for llm alignment under distribution shifts (NeurIPS 2025 · similarity 0.92)
• Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment (NeurIPS 2024 · similarity 0.92)