Research map / Image generation / Text-to-image diffusion
Personalized Image Editing in Text-to-Image Diffusion Models via Collaborative Direct Preference Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes C-DPO, a DPO variant with a GraphSAGE user-preference graph regularizer for personalized image editing, where an LLM generates user-specific edit instructions that FLUX.1-dev + ControlNet applies, evaluated on a 144K-sample synthetic user preference dataset.
Problem
Text-to-image editing models are generic and ignore individual users' aesthetic preferences; personalized image editing has not been formulated or benchmarked before.
Approach
Encode users as nodes in a user-attribute preference graph embedded by a GraphSAGE GNN, condition an LLM editing-instruction policy on these embeddings, and add a neighbor-weighted DPO term so each user's policy is also regularized toward like-minded users' preference pairs.
Key results
- C-DPO achieves CLIP like+dislike similarity 0.354 vs 0.307 for DPO-User and 0.276 for SFT-Vanilla in the prompt-level evaluation.
- In a 50-participant crowd study, the method wins Q1 (general preference) with 0.616 vs 0.210 for DPO-Vanilla and 0.174 for DPO-User.
- Real-user survey (10 participants) gives personalization 4.60 and disentanglement 4.55 on a 1-5 scale.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add graph-based collaboration to user-aware DPO | DPO-User · Prompt alignment (Like+Dislike) | CLIP similarity: 0.307 $\pm$ 0.071 -> 0.354 $\pm$ 0.068 | Helped |
| add graph-based collaboration to user-aware DPO | DPO-User · Prompt alignment (Likes) | CLIP similarity: 0.286 $\pm$ 0.071 -> 0.306 $\pm$ 0.069 | Helped |
| add graph-based collaboration to user-aware DPO | DPO-User · Prompt alignment (Dislikes) | CLIP similarity: 0.269 $\pm$ 0.076 -> 0.294 $\pm$ 0.071 | Helped |
| use graph-based user embeddings instead of textual prompt engineering (ablation) | Personalized editing · Prompt alignment (Like+Dislike) | CLIP similarity: 0.270 $\pm$ 0.051 -> 0.354 $\pm$ 0.068 | Helped |
| use graph-based user embeddings instead of textual prompt engineering (ablation) | Personalized editing · Prompt alignment (Likes) | CLIP similarity: 0.270 $\pm$ 0.045 -> 0.306 $\pm$ 0.069 | Helped |
| use graph-based user embeddings instead of textual prompt engineering (ablation) | Personalized editing · Prompt alignment (Dislikes) | CLIP similarity: 0.200 ± 0.057 -> 0.294 $\pm$ 0.071 | Helped |
| set collaborative scale to 0.01, reducing neighbor influence (ablation) | C-DPO · Personalized image editing | user alignment | Hurt |
| set collaborative scale to 0.50, emphasizing neighbor preferences (ablation) | C-DPO · Personalized image editing | user alignment | Hurt |
| set collaborative scale to 0.15 (ablation) | C-DPO · Personalized image editing | user alignment | Helped |
| increase neighborhood size from K=5 to K=12 (ablation) | C-DPO · Collaborative neighborhood size | CLIP Liked Similarity: 0.358 $\pm$ 0.065 -> 0.344 $\pm$ 0.069 | Hurt |
Surprising findings
- Increasing the neighborhood size K beyond 5 degrades alignment, and a very high collaboration weight (lambda=0.50) hurts performance.
Limitations
- Evaluation relies mainly on synthetic user profiles, with only a small real-user study of 10 participants.
- New users with no edit history and no close neighbors default to generic editing.
- Depends on off-the-shelf Flux and ControlNet, so their biases propagate to edits.
Experimental setup
Datasets: InstructPix2Pix training set COCO captions (80 categories) synthetic user preference dataset (144K samples)
Benchmarks: synthetic user preference dataset (test split)
Metrics: CLIP-text similarity to liked instructions DINO CLIP-I CLIP-T HPS win rate in user study 1-5 Likert personalization and disentanglement ratings
Baselines: Qwen-VL LLaVA SFT-Vanilla SFT-Pix2Pix DPO-Vanilla DPO-User ViPER BAGEL SEED-Llama Nexus-Gen
Builds on: Direct Preference Optimization Diffusion-DPO GraphSAGE LoRA PPD
Closest papers by meaning
- Personalized Preference Fine-tuning of Diffusion Models (CVPR 2025 · similarity 0.91)
- Energy-Guided Optimization for Personalized Image Editing with Pretrained Text-to-Image Diffusion Models (AAAI 2025 · similarity 0.90)
- DreamSteerer: Enhancing Source Image Conditioned Editability using Personalized Diffusion Models (NeurIPS 2024 · similarity 0.89)
- PaRa: Personalizing Text-to-Image Diffusion via Parameter Rank Reduction (ICLR 2025 · similarity 0.89)
- Textualize Visual Prompt for Image Editing via Diffusion Bridge (AAAI 2025 · similarity 0.89)
- Aesthetic Post-Training Diffusion Models from Generic Preferences with Step-by-step Preference Optimization (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.