Research map / Reinforcement learning / Offline RL
Analytic Energy-Guided Policy Optimization for Offline Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AEPO, which derives a closed-form-approximated intermediate energy for energy-guided diffusion policies via Taylor expansion and Gaussian moment-generating functions, and evaluates it on D4RL offline RL benchmarks.
Problem
Estimating the intermediate energy (log-expectation over posterior actions) in energy-guided diffusion policies for offline RL is intractable, so guidance is inexact.
Approach
Derives an analytic approximation of the intermediate energy log-expectation by linearizing Q around a constant action and using the Gaussian moment-generating function on the Gaussian posterior, then trains a neural intermediate energy network to regress this target, with guidance rescaling for robustness.
Key results
- AEPO achieves a mean normalized score of 87.4 on D4RL Gym-MuJoCo tasks versus 86.6 for QGPO (total 786.7 vs 779.2).
- On Maze2D, AEPO reaches a mean dense score of 127.5 and mean sparse score of 132.3, competitive with TCD and Diffuser.
- On Adroit, AEPO reports mean score 55.6 vs 54.6 for LD (total 667.5 vs 655.5).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AEPO instead of QGPO | D4RL Gym-MuJoCo · D4RL Gym-MuJoCo | mean score: 86.6 -> 87.4 | Helped |
| Use AEPO instead of TCD | D4RL Pointmaze (Maze2D) sparse tasks · D4RL Pointmaze (Maze2D) | mean sparse score: 135.8 -> 132.3 | Hurt |
| Use AEPO instead of DD | D4RL Pointmaze (Maze2D) dense tasks · D4RL Pointmaze (Maze2D) | mean dense score: 61.5 -> 127.5 | Helped |
| Use AEPO instead of QGPO | D4RL Antmaze · D4RL Antmaze | mean score: 78.3 -> 80.6 | Helped |
| Use AEPO instead of LD | D4RL Adroit · D4RL Adroit | mean score: 54.6 -> 55.6 | Helped |
| Use Posterior 1 instead of Posterior 2 for posterior approximation (ablation) | AEPO · D4RL Gym-MuJoCo | holistic performance | Helped |
| Train the Q function with IQL rather than CQL or in-support softmax Q-learning (ablation) | AEPO · D4RL Gym-MuJoCo | holistic performance | Helped |
| Add intermediate guidance (ω = 0.1) rather than use no guidance (ω = 0) (ablation) | AEPO · D4RL Gym-MuJoCo | normalized score | Helped |
| Apply guidance rescaling rather than non-rescaling (ablation) | AEPO · D4RL Gym-MuJoCo | normalized score | Helped |
| Increase guidance degree ω with rescaling rather than without rescaling (ablation) | AEPO · D4RL Gym-MuJoCo | normalized score | Mixed |
| Vary β (ablation) | AEPO · D4RL Gym-MuJoCo | normalized score | No clear effect |
| Set expectile weight τ to 0.7 (ablation) | AEPO · D4RL Gym-MuJoCo | normalized score | Helped |
Surprising findings
- Posterior 2 variance estimate performs worse than Posterior 1 in ablations, which the authors attribute to dataset variance contamination.
Limitations
- Relies on a Taylor-expansion linear approximation of Q and mild assumptions for the posterior covariance.
- Performance gains over strong baselines are small on many tasks (e.g., AntMaze medium-play and large-play lower than QGPO/IDQL).
Experimental setup
Datasets: D4RL Gym-MuJoCo D4RL Maze2D D4RL AntMaze D4RL Adroit
Benchmarks: D4RL Gym-MuJoCo D4RL Maze2D D4RL AntMaze D4RL Adroit
Metrics: normalized score mean score total score
Baselines: QGPO D-QL DiffuserLite HD-DA IDQL AdaptDiffuser Consistency-AC HDMI TCD SfBC DD Diffuser
Builds on: Diffusion-QL QGPO Implicit Q-learning DPM-solver Diffuser
Closest papers by meaning
- Energy-Weighted Flow Matching for Offline Reinforcement Learning (ICLR 2025 · similarity 0.90)
- Offline Transition Modeling via Contrastive Energy Learning (ICML 2024 · similarity 0.90)
- Energy-Guided Diffusion Sampling for Offline-to-Online Reinforcement Learning (ICML 2024 · similarity 0.89)
- Prior-Guided Diffusion Planning for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
- OCEAN-MBRL: Offline Conservative Exploration for Model-Based Offline Reinforcement Learning (AAAI 2024 · similarity 0.88)
- Reward-Consistent Dynamics Models are Strongly Generalizable for Offline Reinforcement Learning (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.