Research map / Reinforcement learning / Policy gradient and actor-critic

ReFPO: Reflow Regularization for Flow Matching Policy Gradients

NeurIPS 2026 · Reinforcement learning › Policy gradient and actor-critic · approach: Model-based RL with guarantees · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ReFPO, which adds an unweighted Reflow (CFM) regularizer to Flow Matching Policy Gradients (FPO) to straighten learned flows, enabling accurate one-step inference and more stable PPO-style training on GridWorld, MuJoCo Playground, and Humanoid tracking.

Problem

Flow-based policies trained with online RL need many ODE steps at inference, and FPO's CFM-based proxy ratio becomes volatile with spikes that destabilize PPO-style updates.

Approach

Interprets FPO's advantage-weighted CFM gradient as an implicit online Reflow process and makes it explicit by adding an unweighted CFM (Reflow) loss on the same on-policy action-noise pairs, a single-line change costing no extra flow evaluations.

Key results

• On MuJoCo Playground (10 tasks, 100M steps), ReFPO* (lambda=0.04) reaches 686 10-step and 690 1-step reward vs FPO 641 and 565, with straightness error 0.0116 vs 0.0475.
• On Humanoid Root-only conditioning, ReFPO*-1step achieves 58.7% success and 70.89 MPJPE vs FPO 55.0% and 73.55.
• Explosion rate (proxy-ratio spikes above exp(3)) drops from 0.00365 for FPO to 0.00189 for ReFPO.

Surprising findings

• Gaussian PPO remains the strongest method under full-joint humanoid conditioning (98.7% success) over flow-based policies.
• Advantage-weighted straightening (applying Reflow only to positive-advantage samples) performs worse than uniform Reflow regularization.

Limitations

• Analysis is local (fixed-batch first-order sensitivity) rather than a full characterization of online RL dynamics.
• Experiments limited to FPO-style flow policies; broader diffusion/flow RL methods not tested.
• Reflow coefficient chosen empirically; Gaussian PPO remains stronger in full-joint humanoid tracking.

Experimental setup

Datasets: MuJoCo Playground DM Control tasks AMASS

Benchmarks: GridWorld 25x25 MuJoCo Playground (10 DM Control tasks) Humanoid Control MoCap tracking (AMASS)

Metrics: episode reward success rate alive duration MPJPE straightness error explosion rate inference latency

Baselines: FPO Gaussian PPO ReinFlow Flow-GRPO

Builds on: FPO PPO Rectified Flow / Reflow Flow-GRPO ReinFlow

Closest papers by meaning

• ReinFlow: Fine-tuning Flow Matching Policy with Online Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
• Flow-GRPO: Training Flow Matching Models via Online RL (NeurIPS 2025 · similarity 0.88)
• Flow-DPPO: Divergence Proximal Policy Optimization for Flow Matching Models (NeurIPS 2026 · similarity 0.87)
• FlowPolicy: Enabling Fast and Robust 3D Flow-Based Policy via Consistency Flow Matching for Robot Manipulation (AAAI 2025 · similarity 0.87)
• Flow-Based Policy for Online Reinforcement Learning (NeurIPS 2025 · similarity 0.86)
• Flow Matching Policy Optimization with Mirror Descent and Entropy Constraints (NeurIPS 2026 · similarity 0.86)