Research map / Reinforcement learning / Policy gradient and actor-critic

Quantile-Coupled Flow Matching for Distributional Reinforcement Learning

NeurIPS 2026 · Reinforcement learning › Policy gradient and actor-critic · approach: Distributional value learning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes FlowIQN, a flow-matching distributional critic that sorts source and Bellman target samples within each minibatch to get a quantile-coupled, Wasserstein-aligned flow loss, and evaluates it on fixed-policy return modelling and OGBench offline RL.

Problem

Standard conditional flow matching critics use independent source-target couplings, so their losses are not Wasserstein-aligned projections of Bellman target return distributions, which breaks the Wasserstein contraction basis of distributional RL.

Approach

Replace independent source-target pairing with sorted (quantile-aligned) minibatch pairing in 1D return space to approximate the monotone optimal transport coupling, yielding a loss that upper-bounds squared W2 error; extend with shortcut models for few-step inference.

Key results

• On fixed-policy return modelling (scene-play task2, 1578 state-action pairs), FlowIQN gives the most accurate flow-based critic, substantially improving over Value Flows while remaining competitive with C51, IQN, and CODAC.
• On OGBench, FlowIQN-R achieves 72±6 on cube-double (vs. 63±4 Value Flows, 64±3 IQN) and 47±10 on puzzle-4x4 (vs. 29±2 IQN).
• FlowIQN-FQL is competitive but does not uniformly beat IQN, CODAC, or Value Flows; e.g., 10±4 on cube-triple vs. 24±2 for IQN.

Surprising findings

• Value Flows performs poorly in the fixed-policy W2 diagnostic while remaining a strong offline RL baseline, showing distributional accuracy does not fully explain downstream performance.
• FlowIQN does not uniformly outperform classical categorical or quantile critics in the fixed-policy diagnostic.

Limitations

• Theory is limited to fixed-policy distributional Bellman evaluation and an idealised population setting; the sorted minibatch loss is only a finite-sample approximation.
• Gains are not uniform across environments and policy extraction rules; FlowIQN-FQL often trails IQN and CODAC.
• Small K bootstrapped target samples per transition may make the empirical coupling noisy.

Experimental setup

Datasets: OGBench scene-play-singletask-task2-v0 OGBench single-task variants (cube-double, cube-triple, puzzle-3x3, puzzle-4x4, scene, antmaze-giant, humanoidmaze-large

Benchmarks: OGBench

Metrics: negative 2-Wasserstein distance to MC return targets IQM average task success

Baselines: BC ReBRAC IQN CODAC FQL floq Value Flows C51

Builds on: Conditional flow matching Flow Q-Learning (FQL) IQN Value Flows Shortcut models

Closest papers by meaning

• Composite Flow Matching for Reinforcement Learning with Shifted-Dynamics Data (NeurIPS 2025 · similarity 0.87)
• Flow Matching for Offline Reinforcement Learning with Discrete Actions (NeurIPS 2026 · similarity 0.87)
• Flow Q-Learning (ICML 2025 · similarity 0.86)
• Designing Reinforcement Learning for Diffusion Models: A Unified Path-Space View (NeurIPS 2026 · similarity 0.86)
• Quantile Geometry Regularization for Distributional Reinforcement Learning (NeurIPS 2026 · similarity 0.86)
• Trust Region Q Adjoint Matching (NeurIPS 2026 · similarity 0.85)