Research map / Image generation / Fast diffusion sampling
PrefPaint: Aligning Image Inpainting Diffusion Model with Human Preference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Aligns a Stable-Diffusion-based image inpainting model (Runway) with human aesthetic preferences via RL fine-tuning against a CLIP-based reward model trained on a 51k-image human-scored inpainting dataset, with a theory-motivated reward-uncertainty weighting.
Problem
Image inpainting diffusion models often produce structurally implausible or visually unappealing fills that do not match human aesthetic preferences, and per-pixel regularization against paired ground truth does not capture this one-to-many setting.
Approach
Train a regression reward model on a newly human-scored inpainting dataset, then fine-tune a pretrained inpainting diffusion model with a trust-region-style RL objective whose per-sample weight is an exponential function of a linear-bandit uncertainty bound on the reward model's embedding (||z||_{V^-1}), amplifying updates on reliable samples.
Key results
- PrefPaint reaches 70.16% WinRate vs Runway with a single sample on outpainting prompts (72.38% on warping) and 93.14% with 10 samples, versus 73.40% for Runway itself at S=3.
- Against BrushNet, PrefPaint achieves reward 0.374 and WinRate 71.27% at S=1 versus 49.49% for BrushNet at S=1.
- In the RL comparison, PrefPaint reaches reward 0.37 and WinRate 71.27% versus ImageReward 0.29 / 65.10% and DPOK 0.32 / 64.59%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune Runway with PrefPaint human-preference reinforcement learning | Runway inpainting model · Outpainting prompts | WinRate (S=1): 50.51 -> 70.16 | Helped |
| fine-tune Runway with PrefPaint human-preference reinforcement learning | Runway inpainting model · Warping prompts | WinRate (S=1): 47.35 -> 72.38 | Helped |
| use reward trustiness-aware adaptive weighting (ablation) | PrefPaint · Image inpainting | Reward: 1.7% | Helped |
| use reward trustiness-aware adaptive weighting (ablation) | PrefPaint · Image inpainting | training efficiency: 106.40% | Helped |
| replace adaptive weighting with linear gamma (-kf + b) (ablation) | PrefPaint · Image inpainting | WinRate: 71.27% -> 60.28% | Hurt |
| replace adaptive weighting with constant gamma (ablation) | PrefPaint · Image inpainting | WinRate: 71.27% -> 65.95% | Hurt |
| use exponential gamma with k=0.065, b=0.90 instead of k=0.050, b=0.70 (ablation) | PrefPaint · Image inpainting | WinRate: 71.27% -> 70.47% | Hurt |
| use exponential gamma plus reciprocal term (k=0.100, {0.10, 0.85}) (ablation) | PrefPaint · Image inpainting | WinRate: 71.27% -> 70.07% | Hurt |
| use exponential gamma plus reciprocal term (k=0.120, {0.80, 0.85}) (ablation) | PrefPaint · Image inpainting | WinRate: 71.27% -> 69.95% | Hurt |
| train the reward model with regression rather than classification (ablation) | PrefPaint reward model · Reward model evaluation | reward accuracy | Helped |
Surprising findings
- Many general text-to-image models (e.g., SD v1.5, Kandinsky) have high ratios of negative reward on inpainting prompts, and BERTScore diverges sharply from human judgments on the same samples.
- A constant amplification factor speeds up training more than the proposed adaptive weighting but reduces WinRate and reward.
Limitations
- Reward model reflects a single group's aggregate preferences rather than individual user tastes, as acknowledged by the authors.
- Reward labels come from a limited set of human experts across 3 criteria; the labeling and uncertainty bound rely on a linear-regression approximation of the reward head.
- Evaluated mainly against Runway-based and reward-based baselines; fine-tuning uses only the in-house reward model.
Experimental setup
Datasets: ADE20K ImageNet KITTI DIV2K
Benchmarks: PrefPaint test set (outpainting and warping prompts) PowerPaint/BrushNet/HdPaint comparison set
Metrics: WinRate predicted reward reward variance T2I reward CLIP score BLIP score Aesthetic score classification accuracy (CA) Inception Score inference time
Baselines: Runway Stable Diffusion v1.5 Stable Diffusion v2.1 SD xl SD xl ++ MAT Palette PowerPaint BrushNet HdPaint ImageReward DPOK
Builds on: DPOK D3PO TRPO/PPO DDIM ImageReward
Closest papers by meaning
- HD-Painter: High-Resolution and Prompt-Faithful Text-Guided Image Inpainting with Diffusion Models (ICLR 2025 · similarity 0.89)
- Towards Better Alignment: Training Diffusion Models with Reinforcement Learning Against Sparse Rewards (CVPR 2025 · similarity 0.88)
- Diffusion Model as a Noise-Aware Latent Reward Model for Step-Level Preference Optimization (NeurIPS 2025 · similarity 0.88)
- David and Goliath: Small One-step Model Beats Large Diffusion with Score Post-training (ICML 2025 · similarity 0.88)
- Reward-Instruct: A Reward-Centric Approach to Fast Photo-Realistic Image Generation (NeurIPS 2025 · similarity 0.88)
- Training Diffusion Models with Reinforcement Learning (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.