Research map / Large language models / Alignment and preferences

Diverse Preference Learning for Capabilities and Alignment

ICLR 2025 · Large language models › Alignment and preferences · approach: Active and token-level preference learning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Soft Preference Learning (SPL), which decouples the entropy and cross-entropy terms of the KL penalty in RLHF/DPO to control output diversity as a global, sequence-level temperature, improving diversity, best-of-N math accuracy, and calibration.

Problem

Alignment algorithms such as RLHF and DPO reduce output diversity and overweight majority preferences, harming representation of diverse views, repeated-sampling problem solving, and logit calibration.

Approach

Split the KL-regularizer into separate entropy (alpha) and cross-entropy-to-reference (beta) terms; the optimal policy becomes pi_ref^(beta/alpha) * p^(1/alpha), so alpha/beta acts as a sequence-level global temperature on the DPO policy that preserves sequence ordering, with a DPO-style offline objective.

Key results

• SPL Pareto-dominates DPO with token-level temperature scaling across all nine diversity-quality metrics and outperforms all sampling methods on six.
• On GSM8K-Hard at best-of-128, SPL outperforms standard DPO by 10% and temperature-scaled DPO by 4%, needing 34% fewer samples to match DPO at best-of-128.
• SPL models with global temperature slightly above 1 attain higher accuracy and lower calibration error than their DPO counterparts on TruthfulQA and MMLU.

Measured effects reported in the paper

Surprising findings

• Token-level temperature scaling above about 1.2 produces mostly nonsensical text, whereas SPL remains coherent at much higher global temperatures.
• Content diversity metric rates degenerate high-temperature outputs as less diverse, creating non-monotonic behavior.

Limitations

• Quality degrades at very high global temperatures, so diversity gains trade off against performance.
• Diversity metrics rely on embeddings and an LLM judge (gpt-4o-mini), and the content-diversity metric is non-monotonic at high temperatures.
• Performance drops at large alpha, which the authors attribute partly to dataset noise and the reliance on preference-majority signal.

Experimental setup

Datasets: HH-RLHF UltraChat Ultrafeedback-200k GSM8K MATH TruthfulQA MMLU Arena-Hard

Benchmarks: Arena-Hard GSM8K MATH TruthfulQA MMLU HH-RLHF held-out reward

Metrics: win-rate average reward reference cross-entropy embedding cosine distance logical disagreement content diversity best-of-N accuracy expected calibration error Brier score accuracy

Baselines: DPO with token-level temperature scaling min-p sampling top-p sampling top-k sampling f-DPO base SFT model

Builds on: DPO RLHF f-DPO Zephyr recipe temperature scaling

Closest papers by meaning

• Leveraging robust optimization for llm alignment under distribution shifts (NeurIPS 2025 · similarity 0.90)
• TODO: Enhancing LLM Alignment with Ternary Preferences (ICLR 2025 · similarity 0.89)
• Group Preference Optimization: Few-Shot Alignment of Large Language Models (ICLR 2024 · similarity 0.89)
• Panacea: Pareto Alignment via Preference Adaptation for LLMs (NeurIPS 2024 · similarity 0.89)
• Reward-Augmented Data Enhances Direct Preference Alignment of LLMs (ICML 2025 · similarity 0.89)
• Robust LLM Alignment via Distributionally Robust Direct Preference Optimization (NeurIPS 2025 · similarity 0.89)