Research map / Reinforcement learning / Offline RL

DeFlow: Decoupling Behavior-Prior Modeling and Value Maximization for Offline Policy Extraction

NeurIPS 2026 · Reinforcement learning › Offline RL · approach: Offline RL sample complexity theory

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DeFlow, which keeps a multi-step flow-matching policy as a behavior prior and trains a small action-conditioned residual with a Lagrangian-adapted penalty to improve Q-values, avoiding backpropagation through the ODE solver.

Problem

Extracting high-value actions from expressive multi-step generative (flow) policies in offline RL is costly because optimizing Q through iterative ODE sampling requires solver backpropagation, while shortcut/one-step distillation can compress multimodal behavior and introduces a sensitive BC trade-o

Approach

Decouple behavior modeling from value maximization: the flow prior is trained only by flow matching, and a lightweight residual module conditioned on both state and the sampled action is trained with DDPG-style Q gradients (stop-gradient through the flow), under an adaptive dual-updated penalty that bounds the mean squared residual.

Key results

• On OGBench cube-double-play, DeFlow reaches 40 ±5 average vs FQL 25 ±6 in the main table's manipulation row; in the full table cube-double task1 is 65 ±6 vs FQL 61 ±9.
• DeFlow is competitive but not uniformly best: SORL is stronger on antmaze-large (89 vs 81) and humanoidmaze-medium (64 vs 48).
• Offline-to-online puzzle-4x4 reaches 100 ±0 after online fine-tuning versus FQL 38 ±52; the residual attribution test raises cube-double task1 from 10 ±2 to 62 ±12 with the residual enabled.

Surprising findings

• Removing action conditioning from the residual drops mixed human/expert performance from 40 to 5 in the ablation, showing the residual must depend on the sampled proposal.
• Residual depth does not monotonically help: one layer beats two on some tasks and deeper residuals do not consistently improve cube-double.

Limitations

• Local residual cannot recover behaviors absent from the data or missed by the prior; proximity to the prior does not certify data support.
• Budget delta and residual capacity require selection; adaptive alpha does not prevent critic exploitation.
• Retains multi-step sampling cost and the SORL comparison is not compute-matched.

Experimental setup

Datasets: OGBench D4RL

Benchmarks: OGBench D4RL

Metrics: success rate normalized return

Baselines: BC IQL ReBRAC FAWAC FBRAC IFQL FQL SORL Cal-QL RLPD

Builds on: Flow Q-Learning (FQL) conditional flow matching floq DDPG SAC log-temperature adaptation

Closest papers by meaning

• Offline Reinforcement Learning via High-Fidelity Generative Behavior Modeling (ICLR 2023 · similarity 0.90)
• Learning Multimodal One-step Flow Policy via Value-weighted Optimal Transport (NeurIPS 2026 · similarity 0.88)
• Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RL (ICML 2024 · similarity 0.88)
• Aligning Flow Map Policies with Optimal $Q$-Guidance (NeurIPS 2026 · similarity 0.88)
• Offline Behavior Distillation (NeurIPS 2024 · similarity 0.88)
• Diffusion Subgoal Planning for Long-Horizon Offline Goal-Conditioned Reinforcement Learning (NeurIPS 2026 · similarity 0.88)