Research map / Reinforcement learning / Policy gradient and actor-critic

FLAG: Flow Policy MaxEnt-RL by Latent Augmented Guidance

NeurIPS 2026 · Reinforcement learning › Policy gradient and actor-critic · approach: Regularized scalable deep value learning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes FLAG, a MaxEnt-RL method that trains a flow policy via EM with local importance sampling on a latent-augmented MDP, using a Gaussian local policy around flow anchor actions and supervised flow-matching distillation.

Problem

Expressive generative (flow/diffusion) policies in MaxEnt-RL trained via importance-weighted supervised learning suffer importance weight collapse in high-dimensional action spaces, limiting scalability and sample efficiency.

Approach

Augment the state with a flow latent variable so that proposal and target distributions are localized to the same region (local IS); use a Gaussian local policy centered on the flow's anchor action and a cross-entropy proxy MaxEnt objective, optimized by an MPO-style EM with CFM distillation of improved action labels.

Key results

• On DMC Dog, FLAG achieves IQM average 0.879 vs. 0.857 for DIME and 0.799 for Gaussian CrossQ after 1M steps.
• On MyoSuite, FLAG achieves IQM average 0.933 success rate, above DIME (0.900) and DACERv2 (0.859).
• FLAG with N=2 samples retains performance on DMC Dog (Dog-run 0.818 vs 0.912 at N=8), showing robustness to small sample budgets.

Surprising findings

• FlowRL achieves near-zero success on MyoSuite (avg 0.017) despite competitive DMC Dog results.
• Gradient-based action-gradient method QSM performs very poorly across DMC Dog tasks (avg 0.101).

Limitations

• The base flow policy plus local Gaussian head is one particular instantiation; a more principled ODE-to-SDE construction is left for future work.
• Theoretical monotonic improvement depends on small covariance and approximation error conditions that are only assumed to hold in practice.
• Experiments use N=8 Q-evaluations per update due to computational cost.

Experimental setup

Datasets: DMC Dog tasks MyoSuite tasks MuJoCo-v5 tasks

Benchmarks: DMC Dog (Stand, Walk, Trot, Run) MyoSuite (Reach-Hard, Obj-Hold-Hard, Key-Turn-Hard, Pen-Twirl-Hard) MuJoCo-v5 (HalfCheetah, Walker2d, Ant, Humanoid)

Metrics: IQM return normalized score success rate

Baselines: SDAC DPMD QVPO MaxEntDP DIPO QSM DACERv2 DIME FlowRL CrossQ

Builds on: MPO SAC CrossQ DIME QVPO

Closest papers by meaning

• When Maximum Entropy Misleads Policy Optimization (ICML 2025 · similarity 0.89)
• DIME: Diffusion-Based Maximum Entropy Reinforcement Learning (ICML 2025 · similarity 0.88)
• Maximum Entropy Reinforcement Learning via Energy-Based Normalizing Flow (NeurIPS 2024 · similarity 0.88)
• Maximum Entropy Reinforcement Learning with Diffusion Policy (ICML 2025 · similarity 0.87)
• Random Policy Evaluation Uncovers Policies of Generative Flow Networks (ICML 2025 · similarity 0.87)
• S$2$AC: Energy-Based Reinforcement Learning with Stein Soft Actor Critic (ICLR 2024 · similarity 0.86)