Research map / Image generation / Text-to-image diffusion
D-Fusion: Direct Preference Optimization for Aligning Diffusion Models with Visually Consistent Samples
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes D-Fusion, which builds visually consistent, denoising-trajectory-preserving preference pairs via mask-guided self-attention fusion so DPO/DDPO/DPOK can better align Stable Diffusion with text prompts.
Problem
DPO-based alignment of text-to-image diffusion models is limited because well-aligned and poorly-aligned images differ in unrelated factors (visual inconsistency), and editing methods that produce consistent images lose the per-timestep denoising trajectories needed for RL training.
Approach
Generate a target image from the base image's noise by injecting keys and values from a high-preference reference image inside an auto-extracted cross-attention mask (self-attention fusion) at each denoising step, so the target is well-aligned, visually consistent with the base, and retains its trajectory for DPO.
Key results
- Human raters (22) preferred D-Fusion+DPO images over SD and naive DPO across the three prompt templates.
- Models fine-tuned with DPO+D-Fusion reach higher CLIPScore than naive DPO at equal training data on all three templates.
- DDIM-inversion-based trajectories gave no noticeable alignment improvement, while D-Fusion improved DPO, DDPO and DPOK.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train with D-Fusion-generated visually consistent image pairs instead of naive preference pairs | DPO on Stable Diffusion 2.1-base · Template 1 prompts | CLIPScore | Helped |
| Train with D-Fusion-generated visually consistent image pairs instead of naive preference pairs | DPO on Stable Diffusion 2.1-base · Template 2 prompts | CLIPScore | Helped |
| Train with D-Fusion-generated visually consistent image pairs instead of naive preference pairs | DPO on Stable Diffusion 2.1-base · Template 3 prompts | CLIPScore | Helped |
| Train with D-Fusion-generated visually consistent image pairs instead of naive preference pairs | DPO on Stable Diffusion 2.1-base · Three prompt templates | human preference rate | Helped |
| Use DDIM-inversion trajectories instead of D-Fusion trajectories (ablation) | DPO with visually consistent samples · Template 1 prompts | CLIPScore | No clear effect |
| Use DDIM-inversion trajectories instead of D-Fusion trajectories (ablation) | DPO with visually consistent samples · Templates 2 and 3 prompts | CLIPScore | No clear effect |
| Add D-Fusion-generated visually consistent samples (ablation) | DDPO · Template 1 prompts | CLIPScore | Helped |
| Add D-Fusion-generated visually consistent samples (ablation) | DPOK · Template 1 prompts | CLIPScore | Helped |
| Add D-Fusion-generated visually consistent samples (ablation) | DDPO and DPOK · Templates 2 and 3 prompts | CLIPScore | Helped |
Surprising findings
- Applying self-attention fusion at all layers and timesteps does not produce ideal images; fusion is restricted to U-Net layers 3-6 and timesteps 18-1.
- Masking image features (as in MasaCtrl) produces boundary confusion, so masks are applied to keys and values instead.
Limitations
- Fusion sometimes produces failed cases, requiring an extra adoption-threshold verification step.
- Mask thresholds are set per item by manual visual inspection of sampled images.
- Evaluation relies on CLIPScore both as reward and as metric, and uses small prompt sets (40 training prompts per template).
Experimental setup
Datasets: Visual Relation Dataset (VRD)
Benchmarks: custom prompt templates (animal activity, color-material object, object relations)
Metrics: CLIPScore human preference rate
Baselines: Stable Diffusion 2.1-base naive DPO DDIM inversion DDPO DPOK
Builds on: direct preference optimization DDPO DPOK MasaCtrl DDIM
Closest papers by meaning
- Training Diffusion Models with Reinforcement Learning (ICLR 2024 · similarity 0.90)
- Towards Better Alignment: Training Diffusion Models with Reinforcement Learning Against Sparse Rewards (CVPR 2025 · similarity 0.89)
- Curriculum Direct Preference Optimization for Diffusion and Consistency Models (CVPR 2025 · similarity 0.89)
- Using Human Feedback to Fine-tune Diffusion Models without Any Reward Model (CVPR 2024 · similarity 0.89)
- Inference-Time Alignment of Diffusion Models with Direct Noise Optimization (ICML 2025 · similarity 0.89)
- Direct Distributional Optimization for Provable Alignment of Diffusion Models (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.