Research map / Reinforcement learning / Offline RL
Flow Q-Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Flow Q-learning (FQL), which trains a BC flow-matching policy alone and distills it into a one-step policy that maximizes Q values, avoiding backpropagation through time for offline and offline-to-online RL.
Problem
Policy extraction for expressive flow/diffusion policies in offline RL is hard because maximizing value through iterative generative sampling requires unstable, costly backpropagation through time, and existing extraction schemes are limited or expensive at inference.
Approach
Keep the flow policy trained only with behavioral cloning and train a separate one-step policy that maximizes Q via reparameterized gradients while distilling from the full multi-step flow policy's output; the distillation term is shown to upper-bound a squared 2-Wasserstein behavioral regularizer.
Key results
- FQL achieves the best or near-best performance on most of the 73 state- and pixel-based OGBench and D4RL tasks, with 84% on D4RL antmaze-large-play.
- FQL outperforms the flow-policy variants FBRAC (backprop through time), FAWAC and IFQL on the 50 state-based OGBench tasks, using the same codebase and architecture.
- FQL is one of the fastest flow-based offline RL methods in training and inference, being faster than FBRAC in training and IFQL in inference.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use one-step guidance: maximize Q with a one-step policy distilled from a BC flow policy instead of steering the iterative flow policy | Flow-based offline RL · 50 state-based OGBench tasks | score | Helped |
| Replace one-step guidance with backpropagation through time (ablation) | FQL · 50 state-based OGBench tasks | score | Hurt |
| Tune the BC coefficient α for each task (ablation) | FQL · Three ablation tasks | score | Helped |
| Use the mean rather than the minimum of the two Q values for critic target aggregation (ablation) | FQL · Representative ablation tasks | score | Helped |
| Vary the number of Euler flow steps (ablation) | FQL · Representative ablation tasks | score | No clear effect |
| Replace uniform flow-time sampling with beta or logit-normal sampling (ablation) | FQL · Representative ablation tasks | score | No clear effect |
Surprising findings
- Not using clipped double Q-learning (mean target) often leads to better performance than the minimum, contrary to the common default.
Limitations
- Requires numerically solving the ODE during training to compute the distillation target.
- No built-in exploration mechanism, so it underperforms on puzzle-4x4 online fine-tuning.
- Not evaluated on real-world robotic tasks.
Experimental setup
Datasets: OGBench D4RL
Benchmarks: OGBench D4RL antmaze D4RL adroit
Metrics: success rate normalized return run time
Baselines: BC IQL ReBRAC IDQL SRPO Consistency-AC FAWAC FBRAC IFQL Cal-QL RLPD
Builds on: Behavior-regularized actor-critic Flow matching Diffusion-QL TD3+BC ReBRAC
Closest papers by meaning
- Towards Efficient and Expressive Offline RL via Flow-Anchored Noise-conditioned Q-Learning (ICML 2026 · similarity 0.93)
- Direct Flow Q-Learning (ICML 2026 · similarity 0.93)
- Flow-Based Policy for Online Reinforcement Learning (NeurIPS 2025 · similarity 0.91)
- Flow Matching for Offline Reinforcement Learning with Discrete Actions (NeurIPS 2026 · similarity 0.91)
- Aligning Flow Map Policies with Optimal $Q$-Guidance (NeurIPS 2026 · similarity 0.90)
- Scaling Offline RL via Efficient and Expressive Shortcut Models (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.