Research map / Reinforcement learning / RL from feedback

Feedback Efficient Online Fine-Tuning of Diffusion Models

ICML 2024 · Reinforcement learning › RL from feedback · approach: RL fine-tuning of generative models · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SEIKO, an online RL fine-tuning method for pre-trained diffusion models that interleaves reward-model learning with KL-regularized, optimistic diffusion updates, with a regret bound and experiments on GFP proteins, Stable Diffusion aesthetics, and molecules.

Problem

Fine-tuning pre-trained diffusion models to maximize a reward (e.g., aesthetics, bioactivity) when ground-truth reward queries are expensive, while staying within the feasible (valid) region of the design space.

Approach

Iteratively collect samples from the current diffusion model, query rewards, learn a reward model with an uncertainty (optimism) bonus, and update the diffusion model via a KL-regularized control problem (relative to both the pre-trained and previous model) that has an explicit product-form solution keeping support within the pre-trained model's support.

Key results

• On GFP protein fine-tuning, SEIKO Bootstrap reaches value 3.87±0.01 vs 3.66±0.03 for non-adaptive and 3.63±0.04 for online PPO with 2000 queries.
• On Stable Diffusion aesthetics, SEIKO UCB achieves 8.17±0.33 vs 7.22±0.18 for non-adaptive and 6.13±0.33 for online PPO.
• Theory gives a regret bound whose statistical error term scales as Õ(d/√K) for linear reward models under realizability.

Measured effects reported in the paper

Surprising findings

• Guidance baseline performs worse than the non-adaptive baseline on images (5.78 vs 7.22 aesthetic value).
• On GFP, the Bootstrap variant has lower diversity (2.2) than the UCB variant (2.6) despite higher value.

Limitations

• Image experiments were run only once rather than 10 times, limiting statistical reliability.
• Regret analysis relies on realizability and uncertainty-oracle assumptions holding on the feasible space.
• Molecule experiments are only referenced in a remark rather than detailed in the main text.

Experimental setup

Datasets: GFP fluorescence dataset LAION aesthetics ratings (for aesthetic predictor)

Benchmarks: GFP fluorescence (protein) aesthetic score (Stable Diffusion prompt 'cheetah') QED molecules

Metrics: value J_alpha (expected reward minus KL penalty) diversity (mean Levenshtein distance)

Baselines: Non-adaptive fine-tuning Classifier guidance Online PPO Greedy SEIKO

Builds on: DDPO DPOK classifier guidance PPO linear bandit UCB

Closest papers by meaning

• Adding Conditional Control to Diffusion Models with Reinforcement Learning (ICLR 2025 · similarity 0.90)
• Derivative-Free Guidance in Continuous and Discrete Diffusion Models with Soft Value-based Decoding (NeurIPS 2025 · similarity 0.90)
• Fine-Tuning Discrete Diffusion Models via Reward Optimization with Applications to DNA and Protein Design (ICLR 2025 · similarity 0.89)
• Bridging Model-Based Optimization and Generative Modeling via Conservative Fine-Tuning of Diffusion Models (NeurIPS 2024 · similarity 0.89)
• Directly Fine-Tuning Diffusion Models on Differentiable Rewards (ICLR 2024 · similarity 0.89)
• Training Diffusion Models Towards Diverse Image Generation with Reinforcement Learning (CVPR 2024 · similarity 0.88)