Research map / Large language models / Alignment and preferences

Leveraging robust optimization for llm alignment under distribution shifts

NeurIPS 2025 · Large language models › Alignment and preferences · approach: Robust and calibrated DPO · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DoRA, a method-agnostic distribution-aware KL-DRO objective that reweights each preference sample by a classifier-derived calibration term estimating its alignment with the human-preferred distribution, to handle synthetic-data response mixture shift.

Problem

Preference alignment with LLM-generated synthetic data introduces a mixture response shift, where training responses only partly come from the human-preferred target distribution, biasing learned policies away from true human preferences.

Approach

Train per-source BERT classifiers to estimate whether a response comes from the golden distribution, convert them to importance-weight-style calibration terms under a known mixture model, and multiply each sample's loss by this term inside a KL-DRO log-sum-exp objective, so the robust objective focuses on target-aligned data while avoiding over-pessimism.

Key results

• On HH-RLHF with Mistral-7B, DPO w/ DoRA reaches 75.4 win and 21.4 lose versus DPO 74.8/23.5 and Robust DPO 77.8/20.0, with DoRA achieving the lowest lose rates among pairwise methods.
• In the listwise setting, DoRA improves DPO_PL, RRHF and LIRE on HH-RLHF and summarization; e.g., LIRE on HH with Mistral-7B rises from 72.8 to 84.0 win rate.
• On AlpacaEval 2 LC win rate, LIRE improves from 28.74 to 31.28 with DoRA, while DPO_PL slightly drops (18.80 to 18.27).

Measured effects reported in the paper

Surprising findings

• DoRA slightly decreases DPO_PL performance on AlpacaEval 2 with UltraFeedback, which the authors attribute to chosen responses in UltraFeedback coming from mixed sources violating the single-source assumption.
• The DRO-only variant does not consistently beat the reweighting-only variant; effects depend strongly on the base loss and task (e.g., LIRE is highly sensitive).

Limitations

• Requires coarse source labels (human vs model generated) and an approximately known target/golden distribution, which may be unavailable in open-domain or legacy datasets.
• Assumes each chosen response comes from a single source, which is violated in UltraFeedback and degrades results for DPO_PL.
• Uses reward models as proxies for human preference; the golden distribution is approximated by trusted or GPT-4-quality data.

Experimental setup

Datasets: HH-RLHF Summarization (OpenAI summarize_from_feedback) UltraFeedback

Benchmarks: HH-RLHF Summarization AlpacaEval 2 Arena-Hard

Metrics: win rate lose rate length-controlled win rate

Baselines: DPO R-DPO EXO SimPO DPO_PL RRHF LIRE Robust DPO Dr.DPO RSO

Builds on: DPO KL-DRO Plackett-Luce ranking loss (DPO_PL) RRHF LIRE

Closest papers by meaning

• Robust LLM Alignment via Distributionally Robust Direct Preference Optimization (NeurIPS 2025 · similarity 0.95)
• Panacea: Pareto Alignment via Preference Adaptation for LLMs (NeurIPS 2024 · similarity 0.93)
• Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment (NeurIPS 2024 · similarity 0.92)
• TODO: Enhancing LLM Alignment with Ternary Preferences (ICLR 2025 · similarity 0.92)
• Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment (ICLR 2025 · similarity 0.92)
• Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimization (ICLR 2025 · similarity 0.92)