Research map / Reinforcement learning / RL from feedback
ReinFlow: Fine-tuning Flow Matching Policy with Online Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ReinFlow, an online PPO-based RL method that fine-tunes flow matching policies (Rectified Flow and Shortcut Models) by injecting learnable Gaussian noise to make the denoising chain a discrete-time Markov process with exact likelihoods, enabling fine-tuning at 1-4 denoising steps.
Problem
Flow matching policies are imitation-learned from mixed-quality data and lack exploration; their deterministic ODE sampling gives no tractable action likelihood for policy gradient RL, and discretization error grows at very few denoising steps.
Approach
Inject a learnable noise network into each Euler denoising step so the flow becomes a discrete-time Markov process with Gaussian transitions; this gives exact log-likelihoods for any number of steps, enabling a PPO policy gradient over the full denoising chain, with the noise net discarded after training.
Key results
- Rectified Flow policies gain an average 135.36% net reward growth after ReinFlow fine-tuning in legged locomotion tasks, with 82.63% less wall time than DPPO.
- Shortcut Model policies gain an average 40.34% success-rate increase in state and visual manipulation tasks at four or one denoising steps, comparable to fine-tuned DDIM policies.
- Across three Robomimic visual tasks, ReinFlow-S and ReinFlow-R improve pre-trained success rate by an average of 45.77%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune Rectified Flow policies with online RL | Pre-trained Rectified Flow policies · Locomotion tasks | episode reward: 135.36% | Helped |
| Fine-tune Shortcut Model policies with online RL | Pre-trained Shortcut Model policies · State and visual manipulation tasks | success rate: 40.34% | Helped |
| Condition the noise network on observations and time rather than only observations (ablation) | ReinFlow noise injection network · Ant and Kitchen-partial | success rate | Helped |
| Use moderate noise rather than small noise (ablation) | ReinFlow · Ant and Humanoid locomotion | reward: up to three times higher rewards | Helped |
| Reduce noise magnitude (ablation) | ReinFlow · Visuomotor policies, precision-critical tasks, longer denoising chains, and weakly pre-trained models | performance | Helped |
| Add entropy regularization instead of constraining the policy with W2 regularization (ablation) | ReinFlow · Locomotion tasks | reward | Helped |
| Reduce the W2 regularization coefficient (ablation) | ReinFlow · Locomotion tasks | reward | Helped |
| Increase pre-training data quantity or inference steps (ablation) | Pre-trained flow policies · Hopper-v2 and Square | reward | No clear effect |
| Fine-tune policies with ReinFlow across pre-training data scales (ablation) | Pre-trained flow policies · Hopper-v2 and Square | success rate or reward | Helped |
| Use beta time sampling rather than the default time sampling distribution (ablation) | Pre-trained flow policy fine-tuned with ReinFlow · One-denoising-step inference | performance | Helped |
Surprising findings
- Scaling pre-training data or denoising steps does not consistently improve reward, but RL fine-tuning consistently helps regardless of scale.
- The official FQL implementation lets gradients flow into the multi-step policy during one-step distillation, which can interfere with pretraining; the authors found multi-step FQL struggles to match BC reward.
Limitations
- On-policy PPO design is sample-inefficient, which matters for real-world RL.
- Performance is sensitive to noise magnitude, requiring manual tuning of noise bounds.
- Experiments use relatively small networks; scaling to large flow-based VLA models is untested.
Experimental setup
Datasets: D4RL DPPO behavior cloning datasets Franka Kitchen human teleoperated demonstrations Robomimic (DPPO-processed)
Benchmarks: OpenAI Gym Hopper OpenAI Gym Walker2d OpenAI Gym Ant OpenAI Gym Humanoid Franka Kitchen (complete, mixed, partial) Robomimic Can Robomimic Square Robomimic Transport
Metrics: episode reward success rate wall-clock time denoising steps
Baselines: DPPO FQL DDIM policy DQL IDQL DIPO QSM
Builds on: Rectified Flow Shortcut Models DPPO PPO Flow Q-Learning
Closest papers by meaning
- Aligning Flow Map Policies with Optimal $Q$-Guidance (NeurIPS 2026 · similarity 0.91)
- Flow-Based Policy for Online Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
- Flow Matching for Offline Reinforcement Learning with Discrete Actions (NeurIPS 2026 · similarity 0.89)
- FlowPolicy: Enabling Fast and Robust 3D Flow-Based Policy via Consistency Flow Matching for Robot Manipulation (AAAI 2025 · similarity 0.89)
- ReFPO: Reflow Regularization for Flow Matching Policy Gradients (NeurIPS 2026 · similarity 0.89)
- PRECISE: SDE-Consistent Stochastic Sampling for RL Post-Training of Flow-Matching Models (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.