Research map / Image generation / Text-to-image diffusion
Personalized Preference Fine-tuning of Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Personalizes Stable Cascade text-to-image generation by conditioning on VLM-extracted user embeddings from few-shot preference pairs via decoupled cross-attention, trained with a Diffusion-DPO objective.
Problem
Diffusion-DPO aligns text-to-image models to a single population-level reward, ignoring individual users' preferences, and per-user fine-tuning does not scale or generalize to unseen users.
Approach
Extract user preference embeddings from a few preference examples using a frozen VLM (LLaVA-OneVision), inject them into the diffusion model through added user cross-attention layers (IP-Adapter style), and train with a Diffusion-DPO variant conditioned on user embeddings so one model serves many users and generalizes to new ones.
Key results
- With four preference examples from a new user, PPD achieves an average 76% win rate over Stable Cascade as judged by GPT-4o (abstract).
- On synthetic rewards, PPD wins 60%, 90%, and 93% against Stable Cascade for CLIP, Aesthetic, and HPS respectively.
- On real users, PPD achieves 83% win rate vs Stable Cascade on seen users and 78% on unseen users, aggregated 81%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| condition generation on a user preference embedding and optimize multiple rewards jointly | Stable Cascade · PartiPrompts | CLIP: 31.97 -> 32.66 | Helped |
| condition generation on a user preference embedding and optimize multiple rewards jointly | Stable Cascade · PartiPrompts | Aesthetic: 5.33 -> 5.92 | Helped |
| condition generation on a user preference embedding and optimize multiple rewards jointly | Stable Cascade · PartiPrompts | HPS: 23.87 -> 27.51 | Helped |
| condition generation on user preferences and optimize multiple rewards jointly | Diffusion-DPO · PartiPrompts | CLIP: 32.48 -> 32.66 | Helped |
| condition generation on user preferences and optimize multiple rewards jointly | Diffusion-DPO · PartiPrompts | Aesthetic: 5.46 -> 5.92 | Helped |
| condition generation on user preferences and optimize multiple rewards jointly | Diffusion-DPO · PartiPrompts | HPS: 25.96 -> 27.51 | Helped |
| condition generation on user preferences rather than supervised fine-tuning | SFT · PartiPrompts | CLIP: 32.26 -> 32.66 | Helped |
| condition generation on user preferences rather than supervised fine-tuning | SFT · PartiPrompts | Aesthetic: 5.56 -> 5.92 | Helped |
| condition generation on user preferences rather than supervised fine-tuning | SFT · PartiPrompts | HPS: 25.78 -> 27.51 | Helped |
| use separate Diffusion-DPO models trained for each reward instead of one jointly conditioned model | PPD · PartiPrompts | CLIP: 32.96 -> 32.66 | Hurt |
| use separate Diffusion-DPO models trained for each reward instead of one jointly conditioned model | PPD · PartiPrompts | Aesthetic: 6.42 -> 5.92 | Hurt |
| use separate Diffusion-DPO models trained for each reward instead of one jointly conditioned model | PPD · PartiPrompts | HPS: 28.61 -> 27.51 | Hurt |
Surprising findings
- Linear interpolation of user embeddings at inference trades off rewards even though training used only one-hot single-reward embeddings.
Limitations
- Few-shot example selection is caption-agnostic (random user preferences)
- Evaluation relies on GPT-4o as judge, with reported 83% agreement with ground-truth personalized preferences
- Only explored image generation, not video
Experimental setup
Datasets: Pick-a-Pic v2 PartiPrompts
Benchmarks: PartiPrompts Pick-a-Pic v2 held-out users
Metrics: CLIP score Aesthetic score HPS win rate
Baselines: Stable Cascade Diffusion-DPO SFT Diffusion-DPO (CLIP) Diffusion-DPO (Aesthetic) Diffusion-DPO (HPS)
Builds on: Diffusion-DPO IP-Adapter DPO Stable Cascade
Closest papers by meaning
- Diffusion Model Alignment Using Direct Preference Optimization (CVPR 2024 · similarity 0.93)
- Ranking-based Preference Optimization for Diffusion Models from Implicit User Feedback (NeurIPS 2025 · similarity 0.91)
- Diffusion Model as a Noise-Aware Latent Reward Model for Step-Level Preference Optimization (NeurIPS 2025 · similarity 0.91)
- Smoothed Preference Optimization via ReNoise Inversion for Aligning Diffusion Models with Varied Human Preferences (ICML 2025 · similarity 0.91)
- Personalized Image Editing in Text-to-Image Diffusion Models via Collaborative Direct Preference Optimization (NeurIPS 2025 · similarity 0.91)
- Diffusion-NPO: Negative Preference Optimization for Better Preference Aligned Generation of Diffusion Models (ICLR 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.