Research map / Reinforcement learning / RL from feedback
Advantage Weighted Matching: Aligning RL with Pretraining in Diffusion Models
Shuchen Xue, Chongjian GE, Shilong Zhang, Yichen Li, Zhi-Ming Ma
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use clean-target flow matching with advantage weights instead of DDPO per-step likelihood | AWM vs Flow-GRPO · GenEval Overall (SD3.5M) | overall score: 0.95 -> 0.985; 16.54x speed-up | Improved |
| AWM instead of Flow-GRPO on OCR | SD3.5M · OCR (SD3.5M) | accuracy: 0.89 -> 0.92 with 19.13x speed-up | Improved |
| AWM instead of Flow-GRPO on PickScore | SD3.5M · PickScore (SD3.5M) | PickScore: 23.01 -> 23.11 with 34.67x speed-up | Improved |
| AWM instead of Flow-GRPO on FLUX OCR | FLUX · OCR (FLUX) | accuracy: 0.95 -> 0.95 with 8.53x speed-up | Improved |
| AWM instead of Flow-GRPO on FLUX PickScore | FLUX · PickScore (FLUX) | PickScore: 23.08 -> 23.08 with 6.82x speed-up | No clear effect |
| train AWM longer on SD3.5M OCR | AWM · OCR (SD3.5M) | accuracy: 0.92 -> 0.96 | Improved |
| use 6 Monte Carlo timesteps instead of 4 to estimate the AWM objective (ablation) | AWM · GenEval (SD3.5M) | GenEval score | Improved |
| on-policy rollout instead of EMA rollout with importance sampling (ablation) | AWM · GenEval (SD3.5M) | GenEval score | No clear effect |
| increase KL coefficient beta from 1 to 3 (ablation) | AWM · GenEval (SD3.5M) | GenEval score | Hurt |
| SDE sampling instead of ODE sampling during AWM training (ablation) | AWM · OCR and ImageReward | reward | No clear effect |
| noisy-target denoising score matching instead of clean-target in pretraining (ablation) | EDM · CIFAR-10 and ImageNet-64 | FID | Hurt |
Closest papers by meaning
- Designing Reinforcement Learning for Diffusion Models: A Unified Path-Space View (NeurIPS 2026 · similarity 0.91)
- Reinforcement Learning for Diffusion LLMs with Entropy-Guided Step Selection and Stepwise Advantages (NeurIPS 2026 · similarity 0.90)
- Efficient Online Reinforcement Learning for Diffusion Policy (ICML 2025 · similarity 0.89)
- TMPO: Trajectory Matching Policy Optimization for Diverse and Efficient Diffusion Alignment (NeurIPS 2026 · similarity 0.89)
- DACA-GRPO: Denoising-Aware Credit Assignment for Reinforcement Learning in Diffusion Language Models (NeurIPS 2026 · similarity 0.89)
- Training Diffusion Models with Reinforcement Learning (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
