Research map / Reinforcement learning / Visual and representation RL
Pessimistic Backward Policy for GFlowNets
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PBP-GFN, which trains the GFlowNet backward policy to maximize the likelihood of observed trajectories so the observed backward flow matches the true reward, reducing under-exploitation of high-reward objects.
Problem
GFlowNets trained with flow matching on partially observed trajectories tend to under-exploit high-reward objects, because unobserved backward flow is unconstrained and the forward policy assigns too little probability to them.
Approach
Train the backward policy with a negative log-likelihood objective over replay-buffer trajectories so that observed backward flow approaches the true reward, shifting unobserved backward flow onto observed trajectories while preserving total flow and asymptotic optimality of flow matching.
Key results
- PBP-GFN captures all modes in the hyper-grid and converges to the target Boltzmann distribution faster than TB, uniform, and MaxEnt baselines.
- On molecular generation, PBP-GFN finds more modes and higher top-100 scores while keeping low pairwise Tanimoto similarity.
- PBP-GFN improves over baselines across the four RNA sequence tasks and combines with local search and reward-prioritized buffers to further improve performance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a pessimistic backward policy instead of conventional backward-policy training | PBP-GFN · Hyper-grid | L1 distance to target Boltzmann distribution | Helped |
| Use a pessimistic backward policy instead of TB or MaxEnt | Bag generation · Bag generation | performance | Helped |
| Use a pessimistic backward policy instead of prior backward-policy designs | PBP-GFN · Maximum independent set | average reward and maximum reward | Helped |
| Use a pessimistic backward policy instead of prior backward-policy designs | PBP-GFN · Molecular generation | average score of unique top-100 molecules | Helped |
| Use a pessimistic backward policy instead of prior backward-policy designs | PBP-GFN · Molecular generation | number of modes | Helped |
| Use a pessimistic backward policy instead of prior backward-policy designs | PBP-GFN · Molecular generation | average pairwise Tanimoto similarity | Helped |
| Use a pessimistic backward policy instead of the considered baselines | PBP-GFN · TFBind8, RNA-A, RNA-B, and RNA-C | average score of unique top-100 sequences | Helped |
| Use a pessimistic backward policy instead of the considered baselines | PBP-GFN · TFBind8, RNA-A, RNA-B, and RNA-C | number of modes | Helped |
| Combine PBP-GFN with local search (ablation) | PBP-GFN · RNA-C | performance | Helped |
| Combine PBP-GFN with a reward-prioritized buffer (ablation) | PBP-GFN · RNA-C | performance | Helped |
| Use PBP-GFN alone rather than local search or a reward-prioritized buffer (ablation) | PBP-GFN · RNA-C | performance | No clear effect |
Surprising findings
- Flow matching with partially observed trajectories can assign higher probability to a lower-reward object, and additional sampling cannot correct this because unobserved trajectories receive zero forward probability.
Limitations
- Trades exploitation against exploration; may reduce diversity in environments where exploration is significant.
- Theoretical error bound is derived in an idealized setting; the result is an upper-bound argument rather than a full convergence proof.
- Requires an additional backward-policy training step per round, which adds computation.
Experimental setup
Datasets: hyper-grid bag generation maximum independent set graphs fragment-based molecule generation TFBind8 RNA-Binding (RNA-A, RNA-B, RNA-C)
Benchmarks: hyper-grid bag generation maximum independent set fragment-based molecule generation TFBind8 RNA-Binding
Metrics: L1 distance to Boltzmann distribution number of modes average top-100 reward Tanimoto diversity relative mean error 2-hamming ball modes
Baselines: TB DB uniform backward policy MaxEnt backward policy subTB GFlowNet-AL A2C local search reward-prioritized buffer
Builds on: GFlowNets trajectory balance detailed balance MaxEnt backward policy
Closest papers by meaning
- Looking Backward: Retrospective Backward Synthesis for Goal-Conditioned GFlowNets (ICLR 2025 · similarity 0.94)
- Optimizing Backward Policies in GFlowNets via Trajectory Likelihood Maximization (ICLR 2025 · similarity 0.93)
- Pre-Training and Fine-Tuning Generative Flow Networks (ICLR 2024 · similarity 0.92)
- GFlowNet Training by Policy Gradients (ICML 2024 · similarity 0.92)
- Flow Factorization for Efficient Generative Flow Networks (AAAI 2025 · similarity 0.92)
- Symmetry-Aware GFlowNets (ICML 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.