Research map / Large language models / Alignment and preferences
Direct Density Ratio Optimization: A Statistically Consistent Approach to Aligning Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DDRO, a preference-optimization method that estimates the density ratio between preferred and unpreferred response distributions via Bregman-divergence (logistic) loss, claiming statistical consistency without a Bradley-Terry or prospect-theory preference model.
Problem
Existing LLM alignment methods (RLHF, DPO, KTO) assume a parametric human preference model (Bradley-Terry or prospect theory), which may yield statistically inconsistent estimates where more data does not guarantee convergence to true preferences.
Approach
Reframe alignment as direct density ratio estimation between preferred and unpreferred output distributions, trained with a Bregman divergence (logistic loss instance) plus KL regularization, using unpaired data and a log-sigmoid smoothing function S for stability.
Key results
- On UF-G, DDRO matches or beats KTO and BCO on four of five benchmarks (e.g., LLaMA 7B GSM8K 0.0993 vs 0.0591 KTO); KTO wins AlpacaEval (0.6636 vs 0.5280 for LLaMA 7B).
- On UF-B converted to unpaired form, DDRO is comparable to or better than DPO on most benchmarks, e.g., Pythia 1.4B AlpacaEval 0.6609 vs 0.5785 DPO.
- Log-sigmoid smoothing S(x)=log sigma(x) suppresses gradient spikes seen with the identity S(x)=x (appendix ablation).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train with DDRO instead of BCO on UF-G | LLaMA 7B · BBH (UF-G) | score: 0.3345 -> 0.3330 | Hurt |
| Train with DDRO instead of BCO on UF-G | LLaMA 7B · GSM8K (UF-G) | score: 0.0819 -> 0.0993 | Helped |
| Train with DDRO instead of BCO on UF-G | LLaMA 7B · MMLU (UF-G) | score: 0.3123 -> 0.3182 | Helped |
| Train with DDRO instead of BCO on UF-G | LLaMA 7B · TruthfulQA (UF-G) | score: 0.2166 -> 0.2191 | Helped |
| Train with DDRO instead of KTO on UF-G | LLaMA 7B · AlpacaEval LC Winrate (UF-G) | winrate: 0.6636 -> 0.5280 | Hurt |
| Convert paired UF-B training examples to unpaired and use DDRO instead of DPO | LLaMA 7B · BBH (UF-B) | score: 0.3281 -> 0.3350 | Helped |
| Convert paired UF-B training examples to unpaired and use DDRO instead of DPO | LLaMA 7B · GSM8K (UF-B) | score: 0.1008 -> 0.1001 | Hurt |
| Convert paired UF-B training examples to unpaired and use DDRO instead of DPO | LLaMA 7B · MMLU (UF-B) | score: 0.3332 -> 0.3220 | Hurt |
| Convert paired UF-B training examples to unpaired and use DDRO instead of DPO | LLaMA 7B · TruthfulQA (UF-B) | score: 0.2215 -> 0.2240 | Helped |
| Convert paired UF-B training examples to unpaired and use DDRO instead of DPO | LLaMA 7B · AlpacaEval LC Winrate (UF-B) | winrate: 0.5564 -> 0.5973 | Helped |
| Use a non-identity smoothing function S instead of identity S(x)=x (ablation) | DDRO · Training stability | gradient spikes | Helped |
Surprising findings
- DDRO trained on unpaired data, obtained by discarding pairwise links from UF-B, matches or outperforms paired DPO on most benchmarks.
- Benchmark scores are low overall (e.g., GSM8K around 0.02 for Pythia models), and differences between methods are often small.
Limitations
- Statistical consistency theorem requires finite prompt and response sets and a constant preferred-label probability assumption; the bound includes a misspecification term.
- Practical loss deviates from the theoretical Bregman loss via the log-sigmoid smoothing S, which lacks theoretical justification; DDRO learning rate sensitive to choice of S.
- Evaluation is limited to small Pythia and LLaMA 7B models and uses an LLM judge for AlpacaEval.
Experimental setup
Datasets: ultrafeedback-gpt-3.5-turbo-helpfulness (UF-G) ultrafeedback_binarized (UF-B)
Benchmarks: BBH GSM8K MMLU TruthfulQA AlpacaEval LC Winrate
Metrics: accuracy length-controlled win rate
Baselines: DPO KTO BCO
Builds on: DPO KTO BCO Bregman divergence density ratio estimation (LSIF/KLIEP)
Closest papers by meaning
- ComPO: Preference Alignment via Comparison Oracles (NeurIPS 2025 · similarity 0.93)
- Robust LLM Alignment via Distributionally Robust Direct Preference Optimization (NeurIPS 2025 · similarity 0.92)
- Leveraging robust optimization for llm alignment under distribution shifts (NeurIPS 2025 · similarity 0.92)
- Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment (NeurIPS 2024 · similarity 0.92)
- TODO: Enhancing LLM Alignment with Ternary Preferences (ICLR 2025 · similarity 0.92)
- Panacea: Pareto Alignment via Preference Adaptation for LLMs (NeurIPS 2024 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.