Research map / Reinforcement learning / Policy gradient and actor-critic
GFlowNet Training by Policy Gradients
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Reformulates GFlowNet training as an RL problem with policy-dependent rewards, deriving policy-gradient and TRPO/PPO-style forward training plus a coupled backward-policy design with a convergence and joint-bound analysis.
Problem
GFlowNet training is typically value-based (flow balance / TB / DB objectives), which struggles with exploration in large combinatorial spaces with isolated reward modes; the paper asks whether GFlowNet training can be cast as policy-based RL and how backward policy design can be jointly optimized.
Approach
Define policy-dependent rewards (log ratios of forward and backward policies) so that the trajectory-balance objective equals minimizing expected accumulated reward (a KL divergence) in an absorbing MDP over a DAG; this enables policy-gradient, TRPO-, and PPO-style updates, and backward policy design is cast as a second RL problem solved jointly with forward training.
Key results
- On 256x256 and 128x128 hyper-grids, policy-based methods (RL-G, RL-T) converge faster and to lower TV distance than all value-based baselines (5 runs).
- On SIX6 and QM9, RL-T shows the fastest convergence in mode accuracy, with TB-based and policy-based converged accuracy similar.
- The ablation over lambda shows RL-U with lambda < 1 converges faster than TB-U across all tested settings with better converged TV distance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use policy-based RL training instead of value-based GFlowNet training | GFlowNet training · 256×256 hyper-grid | D_TV convergence | Helped |
| Use RL-G joint training with a guided backward policy | GFlowNet training · 256×256 hyper-grid | D_TV | Helped |
| Use policy-based RL training instead of DB-based training | GFlowNet training · 128×128 hyper-grid | D_TV convergence | Helped |
| Use RL-T trust-region training instead of RL-U | GFlowNet training · Hyper-grid | training stability | Helped |
| Use policy-based training instead of DB-based training | GFlowNet training · SIX6 and QM9 | mode accuracy | Helped |
| Use policy-based training instead of TB-based training | GFlowNet training · SIX6 and QM9 | mode accuracy convergence speed | Helped |
| Use RL-T trust-region training | GFlowNet training · SIX6 and QM9 | convergence speed | Helped |
| Use RL-G guided joint training instead of RL-U with a uniform backward policy | GFlowNet training · SIX6 and QM9 | performance | Helped |
| Vary λ in RL-U; values other than λ=1 (ablation) | RL-U · 256×256 hyper-grid | D_TV convergence speed | Helped |
| Vary λ in RL-U instead of using TB-U with different γ values (ablation) | GFlowNet training · 256×256 hyper-grid | converged D_TV | Helped |
| Vary γ in TB-U (ablation) | TB-U · 256×256 hyper-grid | D_TV | Mixed |
Surprising findings
- Converged accuracy of policy-based methods is similar to TB-based methods on sequence tasks; the gain is mainly in convergence speed.
- The authors' guided backward policy mixing used in prior work lacks theoretical guarantees, which they say their joint RL formulation avoids.
Limitations
- May get trapped in local optima due to gradient estimation variance when the state space is very large.
- Currently restricted to DAG environments; extension to graphs with cycles is future work.
- TRPO trust-region constraint uses a fixed constant zeta_F rather than a schedule.
Experimental setup
Datasets: hyper-grid environment SIX6 PH04 QM9 sEH Bayesian network structure learning data
Benchmarks: hyper-grid 256x256 hyper-grid 128x128 SIX6 QM9 PHO4 sEH Bayesian network structure learning
Metrics: total variation distance Jensen-Shannon divergence mode accuracy
Baselines: DB-U DB-B TB-U TB-B TB-Sub TB-TS
Builds on: Trajectory Balance TRPO PPO Sub-Trajectory Balance torchgfn
Closest papers by meaning
- Optimizing Backward Policies in GFlowNets via Trajectory Likelihood Maximization (ICLR 2025 · similarity 0.94)
- Looking Backward: Retrospective Backward Synthesis for Goal-Conditioned GFlowNets (ICLR 2025 · similarity 0.93)
- Random Policy Evaluation Uncovers Policies of Generative Flow Networks (ICML 2025 · similarity 0.93)
- Pre-Training and Fine-Tuning Generative Flow Networks (ICLR 2024 · similarity 0.92)
- Pessimistic Backward Policy for GFlowNets (NeurIPS 2024 · similarity 0.92)
- Flow Factorization for Efficient Generative Flow Networks (AAAI 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.