Research map / Reinforcement learning / RL from feedback
AlignDiff: Aligning Diverse Human Preferences via Behavior-Customisable Diffusion Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AlignDiff, which trains a transformer attribute-strength model from human pairwise feedback and uses it to relabel locomotion data for an attribute-conditioned DiT diffusion planner that customizes behaviors zero-shot.
Problem
Aligning RL agent behaviors with diverse, abstract, and changing human preferences without retraining whenever preferences shift; prior reward-conditioned and conditional BC/sequence models fail to capture diverse behaviors.
Approach
Quantify relative behavioral attribute strengths from multi-perspective human feedback with a trajectory-level transformer, relabel offline data with these strengths, and train a DiT diffusion planner conditioned on target strengths and attribute masks; at inference, sample candidate trajectories and select the one best matching the preference.
Key results
- AlignDiff achieves average MAE-curve area of 0.525 (synthetic), 0.465 (human labels), and 0.510 (humanness), versus best baseline SM at 0.388, 0.333, and 0.384.
- In a questionnaire with 424 evaluators, AlignDiff average human sorting accuracy is 85.03% with human labels versus 23.12% for the best baseline SM.
- With only 2,000 labels (80% fewer) on Hopper, the MAE-curve area drops only 1.11% (0.628 to 0.621).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AlignDiff rather than the strongest baseline | Preference-conditioned locomotion policies · Hopper, Walker, and Humanoid | MAE curve area: Hopper synthetic: 0.573±0.008 -> 0.628±0.026; Walker synthetic: 0.445±0.012 -> 0.621±0.023; Humanoid synthetic: 0.258±0. | Helped |
| Use AlignDiff rather than the strongest baseline | Preference-conditioned locomotion policies · Hopper, Walker, and Humanoid with human labels | MAE curve area: Hopper: 0.387±0.031 -> 0.485±0.013; Walker: 0.435±0.005 -> 0.597±0.018; Humanoid: 0.272±0.029 -> 0.313±0.015 | Helped |
| Use AlignDiff rather than the strongest baseline | Human-evaluated preference matching · Walker | human sorting accuracy: synthetic labels: 49.49 -> 70.23; human labels: 59.00 -> 93.75 | Helped |
| Use human labels instead of synthetic labels | AlignDiff · Walker | human sorting accuracy: 70.23 -> 93.75 | Helped |
| Add random decision trajectories at 20% noise | AlignDiff training dataset · Walker | MAE curve area: 0.634±0.020 -> 0.559±0.041 | Hurt |
| Add random decision trajectories at 50% noise | AlignDiff training dataset · Walker | MAE curve area: 0.634±0.020 -> 0.384±0.022 | Hurt |
| Reduce synthetic feedback labels from 10,000 to 2,000 | AlignDiff · Hopper | MAE curve area: 0.628±0.026 -> 0.621±0.013 | No clear effect |
| Reduce synthetic feedback labels from 10,000 to 500 | AlignDiff · Hopper | MAE curve area: 0.628±0.026 -> 0.526±0.029 | Hurt |
Surprising findings
- Sorting accuracy was much higher with human labels than with synthetic script labels, suggesting human judgments capture intuition that scripts cannot.
- On the Humanoid benchmark, baselines GC, SM and TDL largely fail to learn useful policies, while AlignDiff is the only method showing preference alignment.
Limitations
- Slow inference due to iterative diffusion sampling.
- Humanness attribute can only be labeled by humans, not synthetic scripts.
- Evaluation limited to locomotion tasks in MuJoCo/DMControl.
Experimental setup
Datasets: Hopper offline dataset (PEDA PPO agents) Walker offline dataset (32 SAC policies) Humanoid offline dataset (40 SAC policies) crowdsourced human feedback labels synthetic script labels
Benchmarks: Hopper Walker Humanoid (MuJoCo / DMControl locomotion) Bisk benchmark (unseen gap-crossing and obstacle tasks, qualitative)
Metrics: area under MAE curve human video sorting accuracy attribute tracking curves distribution coverage p(u,v)
Baselines: GC (goal-conditioned behavior cloning) SM (sequence modeling / Decision Transformer variant) TDL (TD3BC with distilled attribute reward model)
Builds on: RBA Decision Transformer RvS TD3BC DDIM
Closest papers by meaning
- Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning (NeurIPS 2024 · similarity 0.91)
- MaxMin-RLHF: Alignment with Diverse Human Preferences (ICML 2024 · similarity 0.90)
- Direct Alignment with Heterogeneous Preferences (NeurIPS 2025 · similarity 0.89)
- PILAF: Optimal Human Preference Sampling for Reward Modeling (ICML 2025 · similarity 0.89)
- Understanding the Learning Dynamics of Alignment with Human Feedback (ICML 2024 · similarity 0.89)
- Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedback (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.