Research map / Image generation / Text-to-image diffusion
Subject-driven Text-to-Image Generation via Preference-based Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RPO, which fine-tunes only the U-Net of Stable Diffusion for subject-driven generation using Bradley-Terry preference labels from a lambda-Harmonic reward (ALIGN-based) and early stopping on that reward.
Problem
Subject-driven text-to-image generation requires preserving a reference subject while following text prompts, but existing methods (DreamBooth, SuTI, Textual Inversion) are expensive, need many negative samples and gradient steps, and tend to overfit to reference images; there is also no preference
Approach
A weighted harmonic mean of ALIGN image-image and image-text similarity (lambda-Harmonic reward) is used both to generate Bradley-Terry preference labels for a DPO-style loss on a small set of 32 generated images, and as a validation signal for early stopping; only the U-Net is fine-tuned, with a similarity loss added.
Key results
- RPO achieves CLIP-I 0.833 and CLIP-T 0.314 on DreamBench (λ_val=0.5 for CLIP-I), with 400 gradient steps, versus DreamBooth (SD) at 1000 steps with CLIP-I 0.803 and CLIP-T 0.305.
- RPO's DINO score is 0.652, below DreamBooth-Imagen (0.696) and SuTI (0.741), so it does not lead on DINO.
- Ablation: early stopping without the preference loss yields DINO 0.575 and CLIP-T 0.323, while pure similarity loss yields DINO 0.695 and CLIP-T 0.285.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RPO instead of DreamBooth with the SD backbone | DreamBench subject-driven generation · DreamBench | CLIP-I: 0.803 -> 0.833 | Helped |
| Use RPO instead of DreamBooth with the SD backbone | DreamBench subject-driven generation · DreamBench | CLIP-T: 0.305 -> 0.314 | Helped |
| Use RPO instead of DreamBooth with the SD backbone | DreamBench subject-driven generation · DreamBench | DINO: 0.668 -> 0.652 | No clear effect |
| Add preference loss to similarity loss and early stopping (ablation) | RPO · DreamBench | CLIP-T: 0.323 -> 0.329 | Helped |
| Add preference loss to similarity loss and early stopping (ablation) | RPO · DreamBench | 0.3-Harmonic: 0.672 ± 0.013 -> 0.673 ± 0.013 | No clear effect |
| Add preference loss to similarity loss without early stopping (ablation) | RPO · DreamBench | CLIP-T: 0.285 ± 0.027 -> 0.296 ± 0.027 | Helped |
| Add early stopping by the lambda-Harmonic reward (ablation) | RPO with preference loss · DreamBench | CLIP-T: 0.296 ± 0.027 -> 0.329 ± 0.021 | Helped |
| Increase validation lambda from 0.3 to 0.7 (ablation) | RPO · DreamBench | CLIP-I: 0.798 ± 0.039 -> 0.850 ± 0.045 | Helped |
| Increase validation lambda from 0.3 to 0.7 (ablation) | RPO · DreamBench | CLIP-T: 0.329 ± 0.021 -> 0.304 ± 0.023 | Hurt |
| Increase training lambda from 0.0 to 0.7 (ablation) | RPO · DreamBench | CLIP-I: 0.798 ± 0.039 -> 0.831 ± 0.033 | Helped |
| Increase training lambda from 0.0 to 0.7 (ablation) | RPO · DreamBench | CLIP-T: 0.329 ± 0.021 -> 0.314 ± 0.026 | Hurt |
| Replace the harmonic reward with the arithmetic mean (ablation) | RPO validation · DreamBench | CLIP-T: 0.329 ± 0.021 -> 0.318 ± 0.027 | Hurt |
Surprising findings
- Early stopping with the lambda-Harmonic reward trades DINO (subject fidelity) for higher CLIP-T, and the paper notes RPO's CLIP-I/CLIP-T were reported at different lambda_val settings in the main table.
- DPO without the similarity loss achieves high CLIP-T (0.334) but severely overfits, with DINO 0.338.
Limitations
- Context-appearance entanglement, where prompt context alters the subject's appearance.
- Still overfits on some prompts despite regularization.
- Evaluated only on DreamBench, the only open-source subject-driven dataset available to the authors.
Experimental setup
Datasets: DreamBench
Benchmarks: DreamBench
Metrics: DINO CLIP-I CLIP-T lambda-Harmonic reward aesthetic score
Baselines: DreamBooth SuTI Textual Inversion Re-Imagen DisenBooth Custom Diffusion ELITE IP-Adapter SSR-Encoder
Builds on: Diffusion-DPO DreamBooth DPO DDPO
Closest papers by meaning
- Multimodal Large Language Models Make Text-to-Image Generative Models Align Better (NeurIPS 2024 · similarity 0.90)
- Learning to Sample Effective and Diverse Prompts for Text-to-Image Generation (CVPR 2025 · similarity 0.89)
- Boosting Text-to-Video Generative Model with MLLMs Feedback (NeurIPS 2024 · similarity 0.89)
- Preference Adaptive and Sequential Text-to-Image Generation (ICML 2025 · similarity 0.89)
- A User-Friendly Framework for Generating Model-Preferred Prompts in Text-to-Image Synthesis (AAAI 2024 · similarity 0.88)
- Curriculum Direct Preference Optimization for Diffusion and Consistency Models (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.