Research map / Reinforcement learning / Offline RL
Energy-Weighted Flow Matching for Offline Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes energy-weighted flow matching and diffusion that learn energy-guided generative models exactly without auxiliary energy networks, and applies them to offline RL via Q-weighted iterative policy optimization (QIPO).
Problem
Guiding diffusion and flow matching models toward a target distribution q(x) ∝ p(x)exp(-βE(x)) typically requires auxiliary models (intermediate energy or its gradient), back-propagation, or post-processing, and exact energy guidance for general flow matching was not available.
Approach
Derives an energy-guided velocity field from conditional velocity fields and trains it with an energy-weighted flow matching loss, where each data sample's conditional flow target is weighted by softmax of -βE(x0) over the batch, so no intermediate energy model is needed. Extends to diffusion (score) and applies to offline RL by weighting actions by exp(βQ), iteratively refined so the policy impli
Key results
- QIPO-Diff reaches 112.12 ± 0.42 on Medium-Expert Hopper and 97.5 ± 0.53 on AntMaze-umaze default, among the top scores in the D4RL table.
- QIPO-OT averages 86.3 on D4RL locomotion versus 86.6 for QGPO and 87.1 for SRPO, i.e., comparable rather than uniformly superior.
- Action generation time drops from 75.05 ms (QGPO) to 27.26 ms for QIPO-OT, averaged over 1500 runs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use QIPO-Diff instead of the strongest listed baseline | D4RL Medium-Expert Hopper · Medium-Expert Hopper | normalized score: 108.6 -> 112.12 ± 0.42 | Helped |
| Use QIPO-Diff instead of the strongest listed baseline | D4RL Medium-Expert HalfCheetah · Medium-Expert HalfCheetah | normalized score: 97 -> 94.14 ± 0.48 | Hurt |
| Use QIPO-Diff instead of the strongest listed baseline | D4RL Medium-Expert Walker2d · Medium-Expert Walker2d | normalized score: 114.0 -> 110.14± 0.51 | Hurt |
| Use QIPO-OT instead of the strongest listed baseline | D4RL Medium-Expert Hopper · Medium-Expert Hopper | normalized score: 108.6 -> 108.02 ± 5.19 | No clear effect |
| Use QIPO-OT instead of the strongest listed baseline | D4RL Medium Walker2d · Medium Walker2d | normalized score: 86.0 -> 87.61 ± 1.46 | Helped |
| Use QIPO-Diff instead of the strongest listed baseline | D4RL Medium-Replay Walker2d · Medium-Replay Walker2d | normalized score: 85.1 -> 90.08±4.53 | Helped |
| Use QIPO-OT instead of the strongest listed baseline | D4RL Medium-Replay Walker2d · Medium-Replay Walker2d | normalized score: 85.1 -> 78.57 ± 26.09 | Hurt |
| Use QIPO-Diff instead of the strongest listed baseline | D4RL AntMaze-umaze Default · AntMaze-umaze Default | normalized score: 97.1 -> 97.5 ± 0.53 | Helped |
| Use QIPO-Diff instead of the strongest listed baseline | D4RL AntMaze-umaze Diverse · AntMaze-umaze Diverse | normalized score: 85.3 -> 73.88 ± 6.42 | Hurt |
| Use QIPO-OT instead of the strongest listed baseline | D4RL AntMaze-medium Diverse · AntMaze-medium Diverse | normalized score: 84.8 -> 86.42 ± 5.44 | Helped |
| Use QIPO-Diff instead of the strongest listed baseline | D4RL AntMaze-large Play · AntMaze-large Play | normalized score: 66.6 -> 73.25 ± 10.90 | Helped |
| Use QIPO-Diff instead of the strongest listed baseline | D4RL AntMaze-large Diverse · AntMaze-large Diverse | normalized score: 67.9 -> 40.5 ± 20.40 | Hurt |
Surprising findings
- Classifier-free guidance with a scale β≠1 does not generate the intended energy-guided distribution, whereas the energy-weighted approach does.
Limitations
- QIPO is limited to offline RL and does not interact with the environment; online extension is left open.
- Performance is mixed: several tasks (e.g., AntMaze-large Diverse, Walker2d Medium-Replay for QIPO-OT) show high variance or lag baselines.
Experimental setup
Datasets: D4RL
Benchmarks: D4RL
Metrics: normalized return
Baselines: Diffusion-QL QGPO IDQL SRPO Guided Flows SfBC
Builds on: conditional flow matching diffusion models with energy guidance (CEP) QGPO in-support softmax Q-learning classifier-free guidance
Closest papers by meaning
- Analytic Energy-Guided Policy Optimization for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
- Flow Matching for Offline Reinforcement Learning with Discrete Actions (NeurIPS 2026 · similarity 0.89)
- Composite Flow Matching for Reinforcement Learning with Shifted-Dynamics Data (NeurIPS 2025 · similarity 0.89)
- On the Guidance of Flow Matching (ICML 2025 · similarity 0.89)
- Designing Reinforcement Learning for Diffusion Models: A Unified Path-Space View (NeurIPS 2026 · similarity 0.88)
- Energy-Guided Diffusion Sampling for Offline-to-Online Reinforcement Learning (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.