Research map / Reinforcement learning / Policy gradient and actor-critic

Learning Multimodal One-step Flow Policy via Value-weighted Optimal Transport

NeurIPS 2026 · Reinforcement learning › Policy gradient and actor-critic · approach: Model-based RL with guarantees · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes OptiFlow, which distills a multi-step value-aware reference flow policy into a one-step flow policy via state-wise entropic optimal transport with a value-weighted reference marginal, evaluated on OGBench and D4RL.

Problem

Learning efficient one-step flow policies for offline RL that preserve multimodal action distributions without mode collapse or exploiting critic overestimation in out-of-distribution actions.

Approach

Couples one-step policy samples with reference flow-policy samples via entropic OT, where the reference-side marginal is a softmax over critic values (value-weighted) and the cost is action-space distance only; hard row-wise argmax assignments give distillation targets, avoiding direct critic maximization.

Key results

• OptiFlow achieves 48.3±0.4 OGBench state-based average vs 45.6 for GFP and 39.0 for FQL.
• OptiFlow manipulation average 44.5±0.5 vs 40.8 for GFP and 36.3 for FQL.
• OptiFlow achieves best D4RL AntMaze average of 86.6, improving over GFP by 4.0 and FQL by 4.3 points.
• On visual OGBench, OptiFlow averages 75.9 vs 65.4 for FQL and 57.7 for GFP.

Surprising findings

• Replacing the value-weighted reference marginal with a uniform marginal drops the average on the 10-task subset from 57.3 to 9.7, while swapping VaBC for standard BC only drops it to 48.6.
• Adroit performance of OptiFlow (50.5) is not better than the best baselines, being competitive rather than superior.

Limitations

• Additional training-time overhead from multi-sample generation and Sinkhorn OT updates versus pointwise distillation.
• Advantage may be smaller when the reference action distribution is simple or a single local target suffices.
• Dominant cost comes from reference sampling and matching (M samples), not the OT iterations.

Experimental setup

Datasets: OGBench D4RL AntMaze D4RL Adroit

Benchmarks: OGBench D4RL AntMaze D4RL Adroit

Metrics: success rate normalized return

Baselines: BC IQL ReBRAC IDQL IFQL FQL GFP

Builds on: FQL GFP Value-aware behavior cloning Sinkhorn algorithm flow matching

Closest papers by meaning

• Aligning Flow Map Policies with Optimal $Q$-Guidance (NeurIPS 2026 · similarity 0.88)
• DeFlow: Decoupling Behavior-Prior Modeling and Value Maximization for Offline Policy Extraction (NeurIPS 2026 · similarity 0.88)
• Composite Flow Matching for Reinforcement Learning with Shifted-Dynamics Data (NeurIPS 2025 · similarity 0.88)
• Flow-Based Policy for Online Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
• Flow Q-Learning (ICML 2025 · similarity 0.88)
• Flow Matching for Offline Reinforcement Learning with Discrete Actions (NeurIPS 2026 · similarity 0.87)