Research map / Reinforcement learning / RL from feedback
Forward KL Regularized Preference Optimization for Aligning Diffusion Policies
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FKPD, which behavior-clones a diffusion policy from preference-free offline data and then aligns it with preference pairs via a DPO-style loss with forward KL regularization, avoiding reward models and RL.
Problem
Aligning diffusion policies with human preferences without pre-defined reward functions, while avoiding out-of-distribution actions during alignment.
Approach
Derives a DPO objective for diffusion policies approximated via the forward noising chain, and replaces the reverse-KL regularizer of prior DPO methods with a forward-KL term computed as a denoising MSE on the behavior dataset, yielding a single loss on preference pairs plus a regularization term.
Key results
- On MetaWorld with 2.5k dense preferences, FKPD achieves the best success rate on 4 of 6 tasks (e.g., ButtonPress 34.7 vs CPL 24.5).
- In the regularization ablation, FKPD achieves improvement factor 40.0 on Door Open vs 21.7 for RKPD and -42.1 for NRPD.
- FKPD outperforms CPL on 5 of 6 D4RL locomotion tasks, e.g., hopper-medium-replay not in its favor but halfcheetah-expert 94.7 vs 90.9 for CPL.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use FKPD instead of the strongest baseline | MetaWorld 2.5k Dense tasks · MetaWorld 2.5k Dense (six tasks) | success rate: FKPD best on 4 of 6 tasks | Mixed |
| use FKPD instead of the strongest baseline | MetaWorld 20k Sparse tasks · MetaWorld 20k Sparse (six tasks) | success rate: FKPD best on 4 of 6 tasks | Mixed |
| use FKPD instead of CPL | D4RL locomotion tasks · D4RL (six locomotion tasks) | average norm score: FKPD outperforms CPL in 5 out of 6 tasks | Mixed |
| add forward KL regularization rather than reverse KL regularization (ablation) | Diffusion policy alignment · MetaWorld 20k Sparse, Door Open and Drawer Open; Button Press | improvement factor: Door Open: 40.0$\pm{3.5}$ vs 21.7$\pm{8.8}$; Drawer Open: 47.2$\pm{3.7}$ vs 6.9$\pm{3.4}$; Button Press: 58.6$\pm{21.6}$ | Mixed |
| add forward KL regularization rather than reverse KL regularization (ablation) | Diffusion policy alignment · D4RL medium-replay: Walker2d, hopper, halfcheetah | improvement factor: 75.4$\pm{5.0}$ vs -88.7$\pm{11.9}$; 64.8$\pm{12.5}$ vs -33.9$\pm{17.9}$; 14.9$\pm{1.2}$ vs -45.3$\pm{54.3}$ | Helped |
| remove regularization (ablation) | DPO diffusion policy alignment (NRPD) · MetaWorld Button Press and D4RL medium-replay tasks | improvement factor: -100.0$\pm{0.0}$; -95.1$\pm{6.8}$; -98.4$\pm{1.0}$; -100.6$\pm{9.4}$ | Hurt |
| replace forward KL with reverse KL regularization (ablation) | Diffusion policy alignment · D4RL medium-replay tasks | improvement factor: -88.7$\pm{11.9}$; -33.9$\pm{17.9}$; -45.3$\pm{54.3}$ | Hurt |
| replace SFT with FKPD preference optimization | Preference-aligned policy · MetaWorld tasks | success rate | Helped |
Surprising findings
- Without regularization (NRPD), DPO on the diffusion policy causes both winning and losing segment likelihoods to drop sharply, indicating severe OOD behavior.
- Reverse-KL regularized diffusion DPO (RKPD) fails to improve over the pre-trained policy on several D4RL tasks, with negative improvement factors.
Limitations
- Validated only on single-modal (state) observations and single-task settings.
- Regularization term is merged into the sigmoid with a hand-tuned balance factor and bias, adding hyperparameter sensitivity.
Experimental setup
Datasets: MetaWorld offline dataset (CPL) D4RL locomotion datasets
Benchmarks: MetaWorld (BinPicking, ButtonPress, DoorOpen, DrawerOpen, PlateSlide, SweepInto) D4RL walker2d, hopper, halfcheetah (expert, medium-replay)
Metrics: success rate normalized return score improvement factor implicit accuracy
Baselines: SFT P-IQL CPL RKPD NRPD
Builds on: Direct Preference Optimization Diffusion-QL Contrastive Preference Learning Implicit Q-learning Denoising diffusion probabilistic models
Closest papers by meaning
- Designing Reinforcement Learning for Diffusion Models: A Unified Path-Space View (NeurIPS 2026 · similarity 0.90)
- Aligning Diffusion Behaviors with Q-functions for Efficient Continuous Control (NeurIPS 2024 · similarity 0.89)
- Behavior-Regularized Diffusion Policy Optimization for Offline Reinforcement Learning (ICML 2025 · similarity 0.89)
- MPO: An Efficient Post-Processing Framework for Mixing Diverse Preference Alignment (ICML 2025 · similarity 0.89)
- Policy-labeled Preference Learning: Is Preference Enough for RLHF? (ICML 2025 · similarity 0.89)
- TMPO: Trajectory Matching Policy Optimization for Diverse and Efficient Diffusion Alignment (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.