Research map / Reinforcement learning / Offline RL
Adding Conditional Control to Diffusion Models with Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CTRL, which adds new conditional controls to a pre-trained diffusion model by fine-tuning an augmented drift with RL (KL-regularized reward from an offline-trained classifier), evaluated on compressibility and aesthetic-score conditioning with Stable Diffusion v1.5.
Problem
Adding new conditional controls to large pre-trained diffusion models during downstream fine-tuning, when offline datasets with additional labels are small or costly (classifier-free guidance requires large (c,x,y) triplet datasets).
Approach
Frame conditioning as KL-regularized RL over the diffusion SDE: learn an augmented drift with new label embeddings, maximizing a classifier's log-likelihood reward log p(y|x,c) minus KL to the pre-trained model; the optimum yields the target conditional distribution, and the classifier only needs (x,y) pairs when y is independent of c given x.
Key results
- CTRL outperforms all baselines on compressibility and multi-task conditioning by a large margin, per the paper's claims.
- CTRL generates rare class Y=3 (highly aesthetic and high compressibility) with over 90% accuracy in the multi-task setting.
- CTRL achieves better BRISQUE (image quality) and the highest CLIPScore among compared methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CTRL instead of the evaluated baselines for conditioning on compressibility | Stable Diffusion v1.5 · Compressibility-conditioned image generation | classification accuracy | Helped |
| Use CTRL instead of the evaluated baselines for compositional conditioning | Stable Diffusion v1.5 · Multi-task conditional image generation | classification accuracy | Helped |
| Use CTRL for the rare Y=3 compositional condition | Stable Diffusion v1.5 · Multi-task conditional image generation, Y=3 | accuracy | Helped |
| Use CTRL instead of the evaluated baselines | Stable Diffusion v1.5 · Compressibility-conditioned image generation | BRISQUE | Helped |
| Use CTRL instead of the evaluated baselines | Stable Diffusion v1.5 · Compressibility-conditioned image generation | CLIPScore | Helped |
Limitations
- Backpropagation through the full sampling trajectory is memory-intensive, requiring LoRA, gradient checkpointing and truncated backprop.
- Requires a differentiable classifier for direct backprop; non-differentiable rewards need PPO-based methods.
- Classifier errors propagate into the fine-tuned model via the reward.
Experimental setup
Datasets: AVA LAION aesthetic scorer data
Benchmarks: compressibility-conditioned generation multi-task compressibility and aesthetic conditioning
Metrics: classification accuracy macro F1 BRISQUE CLIPScore
Baselines: Reconstruction Guidance Classifier-Free Guidance SMC SVDD MPGD
Builds on: classifier guidance classifier-free guidance DDIM LoRA DPOK
Closest papers by meaning
- Feedback Efficient Online Fine-Tuning of Diffusion Models (ICML 2024 · similarity 0.90)
- Prior-Guided Diffusion Planning for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
- Learning to Sample From Diffusion Models via Inverse Reinforcement Learning (NeurIPS 2026 · similarity 0.89)
- Diffusion Spectral Representation for Reinforcement Learning (NeurIPS 2024 · similarity 0.89)
- Stochastic Control for Fine-tuning Diffusion Models: Optimality, Regularity, and Convergence (ICML 2025 · similarity 0.89)
- Training Diffusion Models with Reinforcement Learning (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.