Research map / Image generation / Fast diffusion sampling
Regularized Conditional Diffusion Model for Multi-Task Preference Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CAMP, which learns multi-task preference representations as conditions for a diffusion planner and adds a mutual-information regularizer to align generated trajectories with those conditions, evaluated on D4RL and Meta-World.
Problem
Return-conditioned diffusion planners need hand-crafted reward functions, which are costly and hard to scale across multiple tasks, and classifier-free guidance often yields generated trajectories inconsistent with their conditions.
Approach
Learn trajectory preference representations (Gaussian, with triplet and KL losses) from multi-task preference pairs, use them as diffusion conditions, and add a variational mutual-information regularizer between noisy trajectories and the representations so generations align with the preferred optimal representation for each task.
Key results
- On D4RL, CAMP averages 71.6 versus 81.6 for IQL (reward-based) and 79.7 for OPPO (re-implemented 59.2), with best scores on halfcheetah-medium-expert, walker2d-medium-expert and hopper-medium-expert (108.9).
- On five unseen Meta-World tasks, CAMP averages 35.8 success versus 17.8 for MTDiff, 19.9 for MT-IQL and 12.4 for MT-BC.
- Ablation removing MI regularization lowers MT-10 success from 68.9 to 67.5 and hopper-medium-expert from 108.9 to 93.8.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add MI regularization to the conditional diffusion model (ablation) | CAMP · walker2d-medium-expert | normalized score: 79.9 ± 22.6 -> 104.8 ± 3.0 | Helped |
| add MI regularization to the conditional diffusion model (ablation) | CAMP · hopper-medium-expert | normalized score: 93.8 ± 6.8 -> 108.9 ± 1.0 | Helped |
| add MI regularization to the conditional diffusion model (ablation) | CAMP · walker2d-medium-replay | normalized score: 56.5 ± 6.7 -> 60.5 ± 1.1 | Helped |
| add MI regularization to the conditional diffusion model (ablation) | CAMP · hopper-medium-replay | normalized score: 54.4 ± 0.1 -> 56.2 ± 0.7 | Helped |
| add MI regularization to the conditional diffusion model (ablation) | CAMP · halfcheetah-medium-expert | normalized score: 94.4 ± 1.4 -> 95.0 ± 2.1 | Helped |
| add MI regularization to the conditional diffusion model (ablation) | CAMP · Meta-World MT-10 | success rate: 67.5 ± 1.7 -> 68.9 ± 2.7 | Helped |
| use a 1-dimensional preference representation instead of the selected 16-dimensional representation (ablation) | CAMP · Meta-World MT-10 | performance | Hurt |
| use a 64-dimensional preference representation instead of the selected 16-dimensional representation (ablation) | CAMP · Meta-World MT-10 | performance | Hurt |
| increase the number of training tasks from 3 to 5 to 10 (ablation) | CAMP · Meta-World five-task evaluation | average success rate: 0.0% -> 4.0% -> 51.2% | Helped |
| condition the diffusion model on CAMP's learned preference representation rather than use reward-conditioned Decision Diffuser | CAMP vs Decision Diffuser · Hopper-medium-expert | alignment between condition and generated trajectory | Helped |
| use CAMP rather than behavior cloning | CAMP vs BC · D4RL Mujoco tasks | normalized score | Helped |
| use CAMP rather than MT-BC, MT-IQL, and MTDiff on unseen tasks | CAMP · five unseen Meta-World tasks | average success rate: 12.4 / 19.9 / 17.8 -> 35.8 | Helped |
Surprising findings
- Decision Diffuser's return conditioning does not produce a smooth relationship between condition and realized return, motivating the alignment regularizer.
- OPPO's reported results were unstable across random seeds in the authors' re-implementation, with degradation on walker2d and hopper.
Limitations
- Average D4RL score (71.6) trails IQL (81.6) that uses true rewards.
- Gains from MI regularization are small on several simple tasks, and performance degrades when the number of tasks grows large.
- Relies on scripted preference labels rather than human preferences.
Experimental setup
Datasets: D4RL halfcheetah/walker2d/hopper (medium, medium-replay, medium-expert) Meta-World MT-10 (SAC replay buffers)
Benchmarks: D4RL Meta-World MT-10
Metrics: normalized return success rate
Baselines: PT OPRL OPPO IQL MTDiff BC MT-BC MT-IQL
Builds on: Decision Diffuser OPPO Preference Transformer MTDiff classifier-free guidance
Closest papers by meaning
- Forward KL Regularized Preference Optimization for Aligning Diffusion Policies (AAAI 2025 · similarity 0.88)
- Calibrated Multi-Preference Optimization for Aligning Diffusion Models (CVPR 2025 · similarity 0.88)
- Sequential Preference Optimization: Multi-Dimensional Preference Alignment with Implicit Reward Modeling (AAAI 2025 · similarity 0.88)
- Reward-Augmented Data Enhances Direct Preference Alignment of LLMs (ICML 2025 · similarity 0.87)
- Personalized Preference Fine-tuning of Diffusion Models (CVPR 2025 · similarity 0.87)
- AlignDiff: Aligning Diverse Human Preferences via Behavior-Customisable Diffusion Model (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.