Research map / Image generation / Text-to-image diffusion
HERO: Human-Feedback Efficient Reinforcement Learning for Online Diffusion Model Finetuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HERO, a human-feedback-efficient online RL framework that fine-tunes Stable Diffusion with learned feedback-aligned embeddings as continuous rewards and feedback-guided initial noise sampling, reaching comparable hand-correction success with about 4x less human feedback than D3PO.
Problem
Online RL fine-tuning of text-to-image diffusion models with human feedback needs thousands of human labels (about 5K for D3PO), and reward models need large datasets that are unavailable for personalized or niche tasks.
Approach
Train an embedding network online with a triplet contrastive loss on human-labeled best/good/bad latents, use cosine similarity to the best sample as a continuous DDPO reward, and sample new initial noises from a Gaussian mixture around previously human-preferred noises.
Key results
- On hand correction, HERO reaches 34.2% normalcy with 1152 feedback versus D3PO's 33.3% at 5K feedback (over 4x more feedback-efficient).
- On four custom tasks, HERO achieves 0.807 (blue-rose), 0.750 (black-cat), 0.912 (narcissus), and 0.995 (mountain) success rates, above all baselines.
- Ablation on narcissus: HERO reaches 0.91 success versus 0.78 for binary rewards and 0.82 for similarity-to-positives.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use HERO with online human feedback instead of D3PO | Stable Diffusion v1.5 on hand deformation correction · Hand deformation correction | normalcy rate: 33.3% -> 34.2% | Helped |
| use HERO instead of the strongest listed baseline | Stable Diffusion v1.5 · blue-rose | success rate: 0.479 (0.030) -> 0.807 (0.115) | Helped |
| use HERO instead of the strongest listed baseline | Stable Diffusion v1.5 · black-cat | success rate: 0.453 (0.142) -> 0.750 (0.130) | Helped |
| use HERO instead of DreamBooth | Stable Diffusion v1.5 · narcissus | success rate: 0.854 (0.092) -> 0.912 (0.007) | Helped |
| use HERO instead of SD-Enhanced | Stable Diffusion v1.5 · mountain | success rate: 0.938 (0.022) -> 0.995 (0.007) | Helped |
| replace continuous similarity rewards with binary rewards (ablation) | HERO on narcissus · narcissus | success rate: 91% -> 78% | Hurt |
| replace learned feedback-aligned representations with SD clean latents (ablation) | HERO on narcissus · narcissus | success rate: 82% -> 76% | Hurt |
| compute reward by similarity to average positive representations instead of the best representation (ablation) | HERO on narcissus · narcissus | success rate: 91% -> 82% | Hurt |
| set best image ratio β=1.0, sampling all images from the best noise latent (ablation) | HERO on black-cat · black-cat | success rate: 70.8% | Mixed |
| fine-tune with DDPO and pretrained PickScore-v1 instead of online human feedback | Stable Diffusion v1.5 · mountain | success rate: 0.412 -> 0.375 | Hurt |
Surprising findings
- DDPO with PickScore-v1 performs worse than the pretrained SD model on the mountain task, which the authors attribute to conflict with general human preference.
Limitations
- Success rates on the custom tasks are judged by a small number of human evaluators, and the tasks are small in number.
- Diversity drops as fine-tuning aligns outputs with human intent.
- Tradeoff controlled by best image ratio beta: higher beta converges faster but may hurt diversity and final accuracy.
Experimental setup
Datasets: online human feedback collected during training
Benchmarks: hand deformation correction blue-rose black-cat narcissus mountain NSFW content safety prompts
Metrics: success rate ImageReward in-batch diversity CLIP score BLIP score content safety rate
Baselines: SD-Pretrained SD-Enhanced DreamBooth D3PO DDPO + PickScore-v1
Builds on: DDPO D3PO DreamBooth Diffusion-DPO
Closest papers by meaning
- David and Goliath: Small One-step Model Beats Large Diffusion with Score Post-training (ICML 2025 · similarity 0.89)
- Using Human Feedback to Fine-tune Diffusion Models without Any Reward Model (CVPR 2024 · similarity 0.88)
- GUIDE: Real-Time Human-Shaped Agents (NeurIPS 2024 · similarity 0.86)
- InstructVideo: Instructing Video Diffusion Models with Human Feedback (CVPR 2024 · similarity 0.86)
- Training Diffusion Models with Reinforcement Learning (ICLR 2024 · similarity 0.86)
- Rich Human Feedback for Text-to-Image Generation (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.