Research map / Reinforcement learning / Offline RL

Diffusion Guidance Is a Controllable Policy Improvement Operator

NeurIPS 2026 · Reinforcement learning › Offline RL · approach: Offline RL sample complexity theory · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CFGRL, which casts policy improvement as classifier-free guidance of a flow-matching policy conditioned on a binary optimality variable, and shows it improves over AWR offline and over goal-conditioned BC without learning value functions.

Problem

Offline RL policy extraction and goal-conditioned behavioral cloning are limited to the performance in the data; weighted regression methods like AWR are unstable and their temperature must be fixed at training time.

Approach

Define policies as products of a reference policy and an optimality factor f(A) (non-negative, non-decreasing), prove that exponentiated attenuation improves the policy, and sample the product via diffusion/flow classifier-free guidance using a single network trained with binary optimality labels, so the guidance weight can be tuned at test time without retraining.

Key results

• On 17 state-based OGBench goal-conditioned tasks, CFGRL with w=3 outperforms GCBC-style baselines on most tasks, e.g., pointmaze-giant 30 vs 2 for GCBC and more than 3x the strongest baseline on some tasks.
• On 9 ExORL tasks, CFGRL beats AWR on most tasks, e.g., walker-stand 782 vs 603 and cheetah-run-backward 262 vs 146.
• On 9 singletask OGBench tasks, CFGRL beats AWR on 5 tasks (e.g., pointmaze-teleport 57 vs 3) but loses on several (e.g., antmaze-large 20 vs 50).

Surprising findings

• For AWR, performance saturates around temperature 1/beta=10, while CFGRL guidance keeps improving beyond that point.
• Performance can decline when guidance is pushed too far, as the policy deviates from the data distribution.

Limitations

• Does not replace full RL; assumes a given value function and does not address how to train it.
• Not state-of-the-art RL on its own; on some offline tasks it underperforms AWR (e.g., antmaze-large-navigate, humanoidmaze-large-navigate).
• Extrapolation beyond the data is limited compared with policy gradient or online RL methods.

Experimental setup

Datasets: ExORL OGBench

Benchmarks: ExORL OGBench

Metrics: success rate return

Baselines: AWR BC Flow BC GCBC Flow GCBC HGCBC Flow HGCBC

Builds on: classifier-free guidance advantage-weighted regression implicit Q-learning OGBench flow matching

Closest papers by meaning

• Prior-Guided Diffusion Planning for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
• Score Regularized Policy Optimization through Diffusion Behavior (ICLR 2024 · similarity 0.89)
• Diffusion Policy Policy Optimization (ICLR 2025 · similarity 0.89)
• Designing Reinforcement Learning for Diffusion Models: A Unified Path-Space View (NeurIPS 2026 · similarity 0.89)
• Diffusion-DICE: In-Sample Diffusion Guidance for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.88)
• Diffusion Subgoal Planning for Long-Horizon Offline Goal-Conditioned Reinforcement Learning (NeurIPS 2026 · similarity 0.88)