Research map / Image generation / Autoregressive and latent generation
Bridging Model-Based Optimization and Generative Modeling via Conservative Fine-Tuning of Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BRAID, a doubly conservative RL-style fine-tuning of pre-trained diffusion models that optimizes an uncertainty-penalized reward model plus a KL penalty to design sequences and images from offline data.
Problem
Offline model-based optimization with pre-trained diffusion models suffers from overoptimization of a learned reward in out-of-distribution regions and generation of invalid designs.
Approach
Fine-tune a pre-trained diffusion model toward exp((r_hat - g_hat)/alpha) p_pre, i.e., a pessimistic uncertainty-penalized reward combined with KL regularization to the pre-trained model, optimized by direct backpropagation; theory gives regret bounds.
Key results
- On DNA/RNA design, BRAID variants outperform STRL, DDOM and Guidance, with fine-tuned models exceeding the best offline value.
- At checkpoint 4 of standard RL fine-tuning on Stable Diffusion, 389/400 generated images are invalid by LLaVA check (mean adjusted score 0.200).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use conservative reward modeling instead of standard reward modeling during diffusion fine-tuning | BRAID vs STRL · Enhancers and 5'UTRs | true reward | Helped |
| Fine-tune the pretrained diffusion model with BRAID rather than conditioning on high-reward classes | BRAID vs DDOM and Offline Guidance · Enhancers and 5'UTRs | true reward | Helped |
| Use BRAID rather than standard RL fine-tuning | Stable Diffusion v1.5 image generation · Image generation | LLaVA-adjusted true reward | Helped |
| Add a conservative uncertainty penalty to standard reward optimization | Diffusion fine-tuning · Image generation | prompt-aligned reward | Helped |
| Continue standard RL fine-tuning to STRL-ckpt-4 | STRL · Image generation (LLaVA-adjusted evaluation) | mean reward: $6.870 $ -> $0.200 $ | Hurt |
| Continue standard RL fine-tuning to STRL-ckpt-4 | STRL · Image generation (LLaVA prompt alignment) | invalid samples / total samples: $0/400$ -> $389/400$ | Hurt |
| Vary the bonus pessimism strength C (ablation) | BRAID-Bonus · Enhancers and 5'UTRs | true reward | No clear effect |
| Vary the bonus parameter C1 while keeping λ=0.1 (ablation) | BRAID-Bonus · Image generation | LLaVA-adjusted true reward | No clear effect |
Surprising findings
- Standard RL fine-tuning raises the learned aesthetic score while most images stop containing the prompted object (389/400 invalid at checkpoint 4).
Limitations
- Theory requires a well-specified reward model and calibrated uncertainty oracle
- Image evaluation relies on a LLaVA judge and a proxy reward rather than ground truth
- Hyperparameter selection uses limited online interactions, departing from a purely offline setting
Experimental setup
Datasets: Enhancers MPRA dataset 5'UTR MPRA dataset AVA
Benchmarks: Enhancers oracle (Enformer-based) UTR oracle LAION Aesthetic Predictor V2 with LLaVA-aided prompt check
Metrics: true reward histogram of generated samples mean reward fraction of invalid images
Baselines: Standard RL (STRL) DDOM Offline Guidance
Builds on: DDPM DDOM Stable Diffusion soft entropy-regularized MDP
Closest papers by meaning
- Fine-Tuning Discrete Diffusion Models via Reward Optimization with Applications to DNA and Protein Design (ICLR 2025 · similarity 0.91)
- Derivative-Free Guidance in Continuous and Discrete Diffusion Models with Soft Value-based Decoding (NeurIPS 2025 · similarity 0.90)
- Reward-Guided Iterative Refinement in Diffusion Models at Test-Time with Applications to Protein and DNA Design (ICML 2025 · similarity 0.89)
- Feedback Efficient Online Fine-Tuning of Diffusion Models (ICML 2024 · similarity 0.89)
- Flow Density Control: Generative Optimization Beyond Entropy-Regularized Fine-Tuning (NeurIPS 2025 · similarity 0.88)
- Improved Off-policy Reinforcement Learning in Biological Sequence Design (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.