Research map / Reinforcement learning / Policy gradient and actor-critic
Flow-Based Policy for Online Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FlowRL, an online actor-critic RL method that parameterizes the policy as a one-step flow-matching velocity field and regularizes it with a Wasserstein-2-style weighted flow-matching loss toward high-value replay-buffer actions.
Problem
Online RL with expressive generative (flow/diffusion) policies suffers from high computational cost, BPTT instability, and misalignment between value-maximizing objectives and generative imitation of buffer data.
Approach
Parameterize the policy as a state-conditioned flow velocity field evaluated with one integration step, and constrain it via a W2 upper bound implemented as advantage-weighted flow matching toward replay-buffer actions (using an expectile-estimated behavior-optimal Q), combined with Q-maximization in a Lagrangian form, avoiding BPTT through the sampler.
Key results
- FlowRL with one flow step matches or outperforms model-free baselines (SAC, TD3, QVPO) on most of 12 DMC-hard/HumanoidBench tasks and 3 DMC-easy/middle tasks across 5 seeds.
- FlowRL achieves comparable results to TD-MPC2 while being much more efficient in wall-clock time.
- Ablation shows the policy constraint improves performance, especially on H1 HumanoidBench tasks; flow-step count (1, 5, 10) has limited effect while increasing steps lengthens BPTT chains.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FlowRL instead of model-free baselines (SAC, QVPO, and TD3) | Online RL on DMControl and HumanoidBench tasks · DMControl and HumanoidBench | returns | Mixed |
| Add the policy constraint (ablation) | FlowRL · DMControl and HumanoidBench tasks | returns | Helped |
| Use fewer flow steps (N=1 rather than N=5 or N=10) (ablation) | FlowRL · DMControl and HumanoidBench tasks | final policy performance | No clear effect |
| Increase the number of flow steps (ablation) | FlowRL · DMControl and HumanoidBench tasks | computational complexity and training time | Hurt |
Surprising findings
- FlowRL performs well without any explicit exploration mechanism, attributed to the stochastic flow actor and exploitation of advantageous actions.
Limitations
- Exploration relies on inherent flow stochasticity; no explicit adaptive exploration mechanism is included.
- The Wasserstein-2 bound relies on a Lipschitz assumption and the constraint is relaxed into a heuristic weighted loss with a constant Lagrange multiplier.
Experimental setup
Datasets: DMControl HumanoidBench
Benchmarks: DMControl HumanoidBench
Metrics: episodic return normalized score wall-clock training time
Baselines: SAC TD3 QVPO TD-MPC2
Builds on: QVPO TD3 SAC flow matching
Closest papers by meaning
- Aligning Flow Map Policies with Optimal $Q$-Guidance (NeurIPS 2026 · similarity 0.91)
- Flow Q-Learning (ICML 2025 · similarity 0.91)
- ReinFlow: Fine-tuning Flow Matching Policy with Online Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
- Flow Matching for Offline Reinforcement Learning with Discrete Actions (NeurIPS 2026 · similarity 0.89)
- Flow-GRPO: Training Flow Matching Models via Online RL (NeurIPS 2025 · similarity 0.89)
- Online Reward-Weighted Fine-Tuning of Flow Matching with Wasserstein Regularization (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.