Research map / Reinforcement learning / RL from feedback
Aligning Language Models with Preferences through $f$-divergence Minimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace forward KL (KL-DPG) with Jensen-Shannon objective (JS-DPG) | f-DPG on lexical constraint target · Lexical constraint (GDC/RLKL) | forward KL divergence and other f-divergences | Helped |
| Use JS-DPG instead of KL-DPG for factual summarization | f-DPG on T5-small summarization · Factual consistency summarization | convergence to target divergence | Helped |
| Use reverse KL (RKL-DPG) instead of forward KL for gender debiasing | f-DPG on gender prevalence constraint · Gender debiasing | constraint matching and entropy | Mixed |
| Use TV-DPG instead of KL-DPG on regard balancing | f-DPG on regard debiasing · Regard balancing (Christians vs Muslims) | regard score convergence | Hurt |
| Use reverse KL objective (RKL-DPG) on scalar sentiment preference | f-DPG on sentiment RLKL target · Sentiment preference | diversity (entropy, Self-BLEU) | Hurt |
| Use JS-DPG objective on scalar sentiment preference | f-DPG on sentiment RLKL target · Sentiment preference | perplexity / divergence across all four f-divergences | Helped |
| Add a baseline (EMA with alpha=0.99 or estimated pseudo-reward mean) to f-DPG gradient | f-DPG training scheme on lexical constraint · Lexical constraint 'amazing' | performance | Helped |
| Scale GPT-2 from small (117M) to xl (1.5B) with f-DPG objectives | f-DPG on sentiment preference · Scaling trends, sentiment | alignment score E[phi(x)] | Helped |
| Scale model size while keeping objective choice fixed | f-DPG across divergence objectives · Scaling trends, sentiment | gap between divergence objectives | No clear effect |
Closest papers by meaning
- Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints (ICLR 2024 · similarity 0.92)
- Towards Efficient Exact Optimization of Language Model Alignment (ICML 2024 · similarity 0.90)
- A Common Pitfall of Margin-based Language Model Alignment: Gradient Entanglement (ICLR 2025 · similarity 0.89)
- Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimization (ICLR 2025 · similarity 0.89)
- Meta-Learning Preferences for Multilingual LLM Alignment (NeurIPS 2026 · similarity 0.89)
- Theoretical Limits of Language Model Alignment (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.