Research map / Reinforcement learning / Policy gradient and actor-critic

Flow Matching Policy Optimization with Mirror Descent and Entropy Constraints

NeurIPS 2026 · Reinforcement learning › Policy gradient and actor-critic · approach: Model-based RL with guarantees · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes FMER, an ODE-based flow matching policy for online RL that uses advantage-weighted conditional flow matching as a simulation-free surrogate for policy mirror descent, plus a closed-form tanh-corrected entropy objective and ESS-based temperature tuning.

Problem

Online continuous-control RL with expressive generative policies faces intractable entropy for SDE-based diffusion policies and unstable, expensive gradients through ODE/denoising chains; existing flow policies lack a principled trust region or closed-form entropy.

Approach

Minimizing an advantage-weighted conditional flow matching loss (weights = softmax of advantages over M candidate actions sampled from the current policy) is claimed to surrogate a policy mirror descent step without backpropagating through the ODE; entropy of the tanh-squashed action distribution is computed in closed form via the divergence of the latent vector field with a tanh correction term,

Key results

• On FrankaKitchen N=1 FMER reaches 80.00% task completion vs 60.00% for DPMD (best baseline) and 40.00% for SAC; N=7 FMER 40.29% vs 23.14% DIPO.
• On MuJoCo-v5 HalfCheetah FMER attains 12332.38 vs 10706.06 for DPMD; Walker 5282.89 is highest; Humanoid 5286.12 beats all generative baselines but trails SAC (5326.73).
• FMER underperforms off-policy baselines on Hopper (2913.45 vs TD3 3662.14), attributed to the entropy constraint sustaining exploration.

Surprising findings

• SAC-Flow collapses on FrankaKitchen (0% for N=1,2,4), underperforming even standard SAC.
• Removing the entropy constraint early (m=0) collapses the policy to near-deterministic behavior, driving tau to zero.

Limitations

• Hopper performance trails off-policy baselines, attributed to entropy constraint over-exploring in a unimodal task.
• Theoretical surrogate relies on approximations (softmax over M candidates as Monte Carlo estimate of Z(s), Gaussian path with sigma to vanishing) and is derived in a sketchy way.
• Entropy divergence estimator is stochastic (Hutchinson), not exact in high dimensions.

Experimental setup

Datasets: FrankaKitchen MuJoCo-v5 (Hopper, HalfCheetah, Walker2d, Humanoid) 2D multi-goal environment

Benchmarks: FrankaKitchen (N=1,2,4,7 task completion rate) MuJoCo-v5 Hopper, HalfCheetah, Walker, Humanoid

Metrics: normalized task completion rate mean maximum return over final 10% of training

Baselines: PPO SAC TD3 DIPO QVPO DPMD SAC-Flow FLAME

Builds on: Conditional Flow Matching Policy Mirror Descent AWR TD3 clipped double Q-learning SAC-Flow

Closest papers by meaning

• PRECISE: SDE-Consistent Stochastic Sampling for RL Post-Training of Flow-Matching Models (NeurIPS 2026 · similarity 0.90)
• Scalable Maximum Entropy Reinforcement Learning for Diffusion Policies via Adjoint Matching (NeurIPS 2026 · similarity 0.90)
• Flow-DPPO: Divergence Proximal Policy Optimization for Flow Matching Models (NeurIPS 2026 · similarity 0.89)
• Composite Flow Matching for Reinforcement Learning with Shifted-Dynamics Data (NeurIPS 2025 · similarity 0.89)
• Efficient Online Reinforcement Learning for Diffusion Policy (ICML 2025 · similarity 0.89)
• TMPO: Trajectory Matching Policy Optimization for Diverse and Efficient Diffusion Alignment (NeurIPS 2026 · similarity 0.89)