Research map / Reinforcement learning / Policy gradient and actor-critic

Flow Matching for Offline Reinforcement Learning with Discrete Actions

NeurIPS 2026 · Reinforcement learning › Policy gradient and actor-critic · approach: Regularized scalable deep value learning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes QDFM, a Q-weighted discrete flow matching method that trains a CTMC over discrete actions for offline RL, extended to preference-conditioned multi-objective and factorized multi-agent settings.

Problem

Offline RL with discrete (and multi-objective, multi-agent) action spaces, where continuous flow-based generative policies are inapplicable because vector fields and action gradients are undefined.

Approach

Replace continuous ODE flows with a continuous-time Markov chain over actions trained by discrete flow matching with a jump-to-endpoint rate field, weighted by softmax-normalized Q-values over model-sampled candidate actions; a factorized rate and path extend it to multi-agent settings, and preference vectors condition the model for multi-objective use.

Key results

• On discretized MuJoCo, QDFM achieves the best score in 4 of 6 tasks (e.g., Hopper expert 17.74±7.61 vs. next best 4.95 for BCQ).
• On Resource Gathering, QDFM reaches HV ratio 0.82 with 4 non-dominated solutions versus 0.67 and 1 for scalarized CQL.
• On the 4-goal gridworld, QDFM attains mode coverage 1.00 while every baseline reaches 0.25.

Surprising findings

• On Deep Sea Treasure, scalarized CQL achieves a higher HV ratio (0.92) than QDFM (0.90), though QDFM recovers 15 non-dominated solutions versus 1.

Limitations

• Inference requires simulating the CTMC over multiple Euler steps, which is slower than single-pass action selection.
• Validation is limited to small gridworlds and does not yet test large discrete action spaces such as language tokens.
• Results on some tasks (e.g., Deep Sea Treasure HV, several MuJoCo and SMAC tasks) are mixed or not best.

Experimental setup

Datasets: Minari MuJoCo (D4RL-style offline datasets) Deep Sea Treasure Resource Gathering D4MORL MO-Hopper SMAC offline datasets (2s3z, 5m_vs_6m)

Benchmarks: Minari MuJoCo discretized (HalfCheetah, Hopper, Walker2d) Deep Sea Treasure Resource Gathering D4MORL MO-Hopper SMAC 2s3z SMAC 5m_vs_6m

Metrics: normalized return hypervolume ratio spacing non-dominated solutions hypervolume win rate action generation time

Baselines: BRAC AWAC AWBC BCQ CQL GreedyQ BoltzQ CFCQL OMAR OMIGA MACQL MODULI

Builds on: Discrete Flow Matching Energy-Weighted Flow Matching QMIX Contrastive Energy Prediction

Closest papers by meaning

• Flow Q-Learning (ICML 2025 · similarity 0.91)
• Aligning Flow Map Policies with Optimal $Q$-Guidance (NeurIPS 2026 · similarity 0.90)
• Composite Flow Matching for Reinforcement Learning with Shifted-Dynamics Data (NeurIPS 2025 · similarity 0.90)
• Designing Reinforcement Learning for Diffusion Models: A Unified Path-Space View (NeurIPS 2026 · similarity 0.90)
• Flow-Based Policy for Online Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
• ReinFlow: Fine-tuning Flow Matching Policy with Online Reinforcement Learning (NeurIPS 2025 · similarity 0.89)