Research map / Image generation / Text-to-image diffusion
UniFL: Improve Latent Diffusion Model via Unified Feedback Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UniFL, a unified feedback-learning framework that improves latent diffusion models (SD1.5, SDXL) in visual quality, aesthetics, and inference speed via perceptual, decoupled aesthetic, and adversarial reward feedback.
Problem
Latent diffusion text-to-image models suffer from structural/visual defects, weak aesthetic appeal, and slow iterative inference, and existing fixes address these separately and are hard to combine.
Approach
Unifies feedback learning across three goals: perceptual feedback from pretrained perception models (instance/semantic segmentation, VGG style) with visual-condition inputs; decoupled aesthetic reward models per dimension (color, layout, detail, lighting) with active prompt selection; and adversarial reward feedback where the reward model is a trainable discriminator that enables few-step generati
Key results
- SD1.5-UniFL at 4 steps reaches FID 33.54 and aesthetic 5.88 vs LCM 42.65 / 5.71 on COCO 5K.
- SDXL-UniFL at 25 steps achieves FID 25.54 and aesthetic 5.98, beating SDXL-Base (27.92 / 5.65) and SDXL-IR (26.71 / 5.81).
- User study: UniFL preferred over ImageReward by 17% and over LCM by 57% for 4-step generation (SDXL).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add PeFL instance-structure feedback (ablation) | UniFL on SD1.5 · User study, structure generation | preference rate: 71.9% vs 28.1% | Helped |
| add style and structure PeFL objectives together (ablation) | UniFL · Style and structure optimization | effectiveness | No clear effect |
| decouple aesthetic rewards by dimension instead of using a global reward (ablation) | SD1.5 aesthetic feedback tuning · User study, aesthetic preference | preference rate: +17% | Helped |
| use actively selected prompts instead of random prompts (ablation) | Decoupled aesthetic feedback learning · User study, aesthetic preference | preference rate: 54.6% vs 45.4% | Helped |
| add adversarial reward-model training (ablation) | UniFL acceleration stage · 4-step versus 20-step inference | user preference: 52.5% vs 47.4% | Helped |
| replace SOLO with Mask2Former as the PeFL instance segmentation model (ablation) | PeFL structure optimization on SD1.5 · User study, structure optimization | performance | No clear effect |
| use a small color-reward hinge coefficient (ablation) | Decoupled aesthetic feedback learning · Color aesthetics | improvement | No clear effect |
| use a large color-reward hinge coefficient (ablation) | Decoupled aesthetic feedback learning · Color aesthetics | image saturation | Hurt |
| disable the adversarial objective while retaining the full reward-optimization timesteps (ablation) | UniFL acceleration stage · Low-step generation | generation quality | Hurt |
| vary inference steps from 1 to 8 (ablation) | UniFL · Inference acceleration | generation quality | Mixed |
| use style PeFL instead of pre-train loss for style optimization (ablation) | SD1.5 and SDXL · Style generation | style response rate: about 15% improvement | Helped |
| use random prompts instead of actively selected prompts (ablation) | Decoupled aesthetic feedback learning · Prompt selection analysis | average nearest embedding similarity: 0.89 vs 0.73 | Hurt |
Surprising findings
- Aesthetic score of SD1.5 rose from 5.26 to 5.88 with adversarial acceleration, i.e., fewer steps gave better aesthetics than the base model.
- Higher-precision instance segmentation (Mask2Former) did not improve PeFL results over SOLO, which the authors attribute to dense vs sparse supervision.
Limitations
- 1-step generation quality is noticeably weaker than SDXL-Turbo.
- Requires a two-stage training pipeline; authors note single-stage simplification as future work.
- Relies on closed-set perception models (SOLO trained on COCO's 80 categories).
Experimental setup
Datasets: COCO2017 DiffusionDB LAION aesthetic subset COCO Stuff in-house human aesthetic preference data
Benchmarks: COCO2017 val 5K
Metrics: FID CLIP score LAION aesthetic score user preference rate
Baselines: ImageReward DreamShaper DPO LCM SDXL-Turbo SDXL-Lightning DreamShaper LCM
Builds on: ReFL (ImageReward) Latent Diffusion Models SDXL-Turbo adversarial diffusion DDIM
Closest papers by meaning
- David and Goliath: Small One-step Model Beats Large Diffusion with Score Post-training (ICML 2025 · similarity 0.90)
- DEEM: Diffusion models serve as the eyes of large language models for image perception (ICLR 2025 · similarity 0.89)
- SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis (ICLR 2024 · similarity 0.89)
- Dual Diffusion for Unified Image Generation and Understanding (CVPR 2025 · similarity 0.89)
- NoiseCLR: A Contrastive Learning Approach for Unsupervised Discovery of Interpretable Directions in Diffusion Models (CVPR 2024 · similarity 0.89)
- FlashEval: Towards Fast and Accurate Evaluation of Text-to-image Diffusion Generative Models (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.