Research map / Reinforcement learning / Visual and representation RL
Normalizing Flows are Capable Models for Continuous Control
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a single normalizing flow architecture (coupling blocks with linear permutation flows and LayerNorm) usable as policy, Q-function, and occupancy measure across imitation, offline RL, goal-conditioned and unsupervised RL.
Problem
Diffusion and autoregressive models used in RL are expressive but computationally heavy or require discrete representations, and normalizing flows are underused in RL due to a belief they lack expressivity.
Approach
A simple conditional normalizing flow block (RealNVP-style affine coupling MLP plus a generalized-permutation linear flow with LU parameterization, LayerNorm in coupling nets) trained by both MLE and variational inference, plugged directly into BC, GCBC, TD3+BC-style offline RL, and UGS with a single network serving as Q-function and coverage density.
Key results
- NF-GCBC exceeds GCBC on 40/45 OGBench tasks and outperforms FM-GCBC by 77% on average.
- NF-RLBC outperforms all baselines on 15/30 offline RL tasks and is 230% better than the next best baseline on puzzle-3x3-play.
- NF-BC uses about 6x fewer parameters than diffusion policies while remaining competitive on imitation learning benchmarks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use more NF blocks in NF-BC (ablation) | NF-BC · Push-T | normalized reward: 0.77 to 0.87 across tested block counts; 48 blocks: 0.87, 54 blocks: 0.85 | Mixed |
| Vary the coupling layer width (ablation) | NF-BC · Push-T | normalized reward: 0.79, 0.82, 0.823 | Mixed |
| Reduce denoising-trick noise from 0.5 to 0.03 (ablation) | NF-BC · Push-T | normalized reward: 0.77 -> 0.873 | Helped |
| Use noise_std 0.0 for the denoising trick (ablation) | NF-BC · Push-T | normalized reward: NaN | Hurt |
| Add permutation flows (ablation) | NF-GCBC · antmaze-large-navigate | success rate: 0.26 -> 0.32 | Helped |
| Add permutation flows (ablation) | NF-GCBC · scene-play | success rate: 0.04 -> 0.42 | Helped |
| Add permutation flows (ablation) | NF-GCBC · cube-single-play | success rate: 0.0 -> 0.795 | Helped |
| Add LayerNorm (ablation) | NF-GCBC · antmaze-large-navigate | success rate: 0.29 -> 0.32 | Helped |
| Add LayerNorm (ablation) | NF-GCBC · scene-play | success rate: 0.3 -> 0.42 | Helped |
| Add LayerNorm (ablation) | NF-GCBC · cube-single-play | success rate: 0.0 -> 0.795 | Helped |
| Add the sampling trick (ablation) | NF-GCBC · antmaze-large-navigate | success rate: 0.3 -> 0.32 | Helped |
| Add the sampling trick (ablation) | NF-GCBC · scene-play | success rate: 0.42 -> 0.42 | No clear effect |
Surprising findings
- Removing permutation flows drops NF-GCBC scene-play from 0.42 to 0.04 and cube-single-play from 0.795 to 0.0.
Limitations
- NFs must be invertible, which restricts architectures.
- The NF architecture itself is adapted from prior work rather than newly designed.
- Results on baselines were taken from prior work without error bars for some IL experiments.
Experimental setup
Datasets: OGBench Push-T Multimodal Ant UR3 BlockPush Kitchen
Benchmarks: OGBench Push-T Multimodal Ant UR3 BlockPush Kitchen
Metrics: normalized reward success rate normalized number of distinct behaviors aggregated final success rate
Baselines: DiffPolicy-C DiffPolicy-T VQ-BeT BeT GCBC FM-GCBC GCIQL QRL CRL ReBRAC IDQL IFQL
Builds on: RealNVP GLOW NICE TD3+BC OGBench
Closest papers by meaning
- Flow Matching for Offline Reinforcement Learning with Discrete Actions (NeurIPS 2026 · similarity 0.88)
- Composite Flow Matching for Reinforcement Learning with Shifted-Dynamics Data (NeurIPS 2025 · similarity 0.88)
- PRECISE: SDE-Consistent Stochastic Sampling for RL Post-Training of Flow-Matching Models (NeurIPS 2026 · similarity 0.87)
- Designing Reinforcement Learning for Diffusion Models: A Unified Path-Space View (NeurIPS 2026 · similarity 0.87)
- ReinFlow: Fine-tuning Flow Matching Policy with Online Reinforcement Learning (NeurIPS 2025 · similarity 0.87)
- Flow-Based Policy for Online Reinforcement Learning (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.