Research map / Reinforcement learning / Policy gradient and actor-critic
Random Policy Evaluation Uncovers Policies of Generative Flow Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that evaluating a fixed uniform random policy with transformed rewards recovers GFlowNet state flows under tree or path-invariant DAG conditions, and proposes Rectified Random Policy Evaluation (RPE) as a simpler training method.
Problem
The relationship between GFlowNets and standard (non-MaxEnt) reinforcement learning, in particular policy evaluation, is largely unexplored, even though both are sequential decision-making frameworks.
Approach
Under tree-structured DAGs (and non-tree DAGs with path-invariant branching ratios), the GFlowNet state flow under flow iteration with uniform backward policy equals the value of a uniform random policy under reward transformed by a branching-ratio factor, up to a scaling g(s). RPE exploits this by fitting flows to scaled policy-evaluation targets for a fixed random policy.
Key results
- On RNA generation, RPE reaches about 99.8-100% accuracy on RNA1-RNA3 and 97.7% on RNA4, versus at most 89.99% for baselines (M-DQN on RNA2).
- RPE discovers 42-90 modes across the four RNA tasks, more than all GFlowNet and MaxEnt RL baselines (e.g., 90 vs 79 for TB-GFN on RNA4).
- On TF Bind, all methods find comparable numbers of modes, and RPE generally outperforms baselines in accuracy by a small margin.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RPE instead of the strongest listed baseline | RNA generation · L14_RNA1 | accuracy: 89.55\pm0.51 -> 99.81\pm0.19 | Helped |
| Use RPE instead of the strongest listed baseline | RNA generation · L14_RNA2 | accuracy: 89.99\pm0.43 -> 100.0\pm3.97 | Helped |
| Use RPE instead of the strongest listed baseline | RNA generation · L14_RNA3 | accuracy: 81.80\pm0.59 -> 100.0\pm2.45 | Helped |
| Use RPE instead of the strongest listed baseline | RNA generation · L14_RNA4 | accuracy: 79.14\pm0.78 -> 97.68\pm3.03 | Helped |
| Use RPE instead of the strongest listed baseline | RNA generation · L14_RNA1 | number of modes: 34\pm2 -> 42\pm2 | Helped |
| Use RPE instead of the strongest listed baseline | RNA generation · L14_RNA2 | number of modes: 31\pm2 -> 46\pm6 | Helped |
| Use RPE instead of the strongest listed baseline | RNA generation · L14_RNA3 | number of modes: 31\pm5 -> 52\pm9 | Helped |
| Use RPE instead of the strongest listed baseline | RNA generation · L14_RNA4 | number of modes: 79\pm7 -> 90\pm6 | Helped |
| Use fixed uniform backward policy instead of learned backward policy (ablation) | TB-GFN, SubTB-GFN, and DB-GFN · TF Bind generation | accuracy | Hurt |
| Use fixed uniform backward policy instead of learned backward policy (ablation) | TB-GFN, SubTB-GFN, and DB-GFN · TF Bind generation | convergence speed | Helped |
Surprising findings
- Fitting the flow of a fixed uniform random policy with transformed rewards matches GFlowNet reward-matching behavior without iterative policy improvement or entropy regularization.
Limitations
- Equivalence for non-tree DAGs requires a path-invariance condition that fails in tasks such as HyperGrid with boundary effects.
- Sampling policy requires separate forward passes per child state, which is computationally costly.
- Effectiveness in environments with very long trajectories is untested and may be limited by local credit assignment.
Experimental setup
Datasets: TFBind (DNA binding reward tasks) RNA binding energy tasks (ViennaRNA) QM9 HOMO-LUMO gap (MXMNet proxy) set generation
Benchmarks: TF Bind generation RNA sequence generation QM9 molecule generation
Metrics: accuracy (relative error of mean reward vs target) number of modes discovered
Baselines: FM-GFN DB-GFN TB-GFN SubTB-GFN Soft DQN Munchausen DQN
Builds on: GFlowNets flow matching detailed balance trajectory balance soft Q-learning
Closest papers by meaning
- Optimizing Backward Policies in GFlowNets via Trajectory Likelihood Maximization (ICLR 2025 · similarity 0.93)
- GFlowNet Training by Policy Gradients (ICML 2024 · similarity 0.93)
- Pre-Training and Fine-Tuning Generative Flow Networks (ICLR 2024 · similarity 0.90)
- Flow Factorization for Efficient Generative Flow Networks (AAAI 2025 · similarity 0.90)
- A Theory of Non-acyclic Generative Flow Networks (AAAI 2024 · similarity 0.90)
- QGFN: Controllable Greediness with Action Values (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.