Research map / Reinforcement learning / Policy gradient and actor-critic

Aligning Flow Map Policies with Optimal $Q$-Guidance

NeurIPS 2026 · Reinforcement learning › Policy gradient and actor-critic · approach: Model-based RL with guarantees · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes flow map policies (learned two-time jump operators for flow-based actors) and a closed-form trust-region Q-guided update (FMQ) plus a renoising beam search (QGBS) for fast offline-to-online RL on OGBench and RoboMimic.

Problem

Generative flow/diffusion policies are expensive at inference because generating each action requires many ODE steps, and existing one-step policies rely on best-of-N heuristics for critic-guided improvement during offline-to-online RL.

Approach

Learn flow map (two-time jump) policies so actions are generated in one step, and derive a closed-form optimal average-velocity target under a trust-region constraint from a first-order critic linearization, used as a self-bootstrapped regression target (FMQ); add a stochastic renoising + Q-guided beam search at inference (QGBS).

Key results

• FMQ reaches IQM success 0.91 [0.89, 0.93] vs 0.75 for MVP (21.3% relative gain) and 0.86 for QC across 12 tasks.
• FMQ + QGBS reaches IQM 0.93 [0.91, 0.94], the highest among compared methods.
• FMQ reaches MVP's best success level about 2.77x faster on average during the online phase (1M-2M steps), up to 6.14x on hmaze-med-task3.

Surprising findings

• QGBS degrades performance on some locomotion tasks (hmaze-med-t3 drops 15.9% for MVP+QGBS as reported), indicating inference-time search is not uniformly beneficial.
• Best-of-N selection is strictly less efficient than renoising with K=1, B=4, M=4 which matches the best IQM with 20 NFE versus 32.

Limitations

• Effectiveness depends on critic accuracy and the local linearity assumed in the first-order Q approximation.
• Evaluated only in simulation; no physical robot deployment.
• QGBS can hurt performance on some locomotion tasks.

Experimental setup

Datasets: OGBench RoboMimic

Benchmarks: OGBench RoboMimic

Metrics: success rate IQM success rate speedup ratio of steps to reach success threshold

Baselines: QC MVP MVP*

Builds on: mean flow policies flow matching clipped double Q-learning Q-chunking MVP

Closest papers by meaning

• Flow-Based Policy for Online Reinforcement Learning (NeurIPS 2025 · similarity 0.91)
• ReinFlow: Fine-tuning Flow Matching Policy with Online Reinforcement Learning (NeurIPS 2025 · similarity 0.91)
• Flow Q-Learning (ICML 2025 · similarity 0.90)
• Flow Matching for Offline Reinforcement Learning with Discrete Actions (NeurIPS 2026 · similarity 0.90)
• Generative Control as Optimization: Time Unconditional Flow Matching for Adaptive and Robust Robotic Control (NeurIPS 2026 · similarity 0.89)
• Trust Region Q Adjoint Matching (NeurIPS 2026 · similarity 0.89)