Research map / Reinforcement learning / Policy gradient and actor-critic
Optimizing Backward Policies in GFlowNets via Trajectory Likelihood Maximization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Trajectory Likelihood Maximization (TLM), an alternating scheme that learns the GFlowNet backward policy by maximum likelihood on forward-sampled trajectories while the forward policy is trained with any GFlowNet or soft-RL loss, and evaluates it on hypergrid, bit sequence, sEH and QM9 tasks.
Problem
Backward policy optimization in GFlowNets is poorly understood; prior soft-RL equivalence holds only for a fixed backward policy, and naively training the backward policy with the forward objective can hurt convergence.
Approach
Alternate between maximizing the likelihood of on-policy forward trajectories under the backward policy (one stochastic gradient step) and optimizing the forward policy as a soft RL problem with rewards induced by the updated backward policy, making the method compatible with any GFlowNet or soft RL algorithm; stability is aided by target networks, decayed learning rates and uniform initialization
Key results
- On the 4-dimensional hypergrid, TLM converges fastest for SoftDQN, MDQN and DB under both standard and hard reward designs.
- On bit sequences, TLM gives a significant mode-discovery improvement for DB over other backward approaches, while all backward approaches find all 60 modes for MDQN.
- On QM9, TLM speeds up mode discovery with DB and yields better Pearson correlation with SubTB, but on sEH it often falls behind other backward approaches.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TLM instead of other backward-policy approaches | SoftDQN, MDQN, and DB · Hypergrid, standard and hard reward designs | L1 distance | Helped |
| Use TLM instead of other backward-policy approaches | SubTB · Hypergrid | L1 distance | Helped |
| Use TLM instead of other backward-policy approaches | TB · Hypergrid, standard reward design | L1 distance | Hurt |
| Use TLM instead of other backward-policy approaches | TB · Hypergrid, hard reward design | L1 distance | No clear effect |
| Use TLM instead of other backward-policy approaches | DB · Bit Sequences | number of discovered modes | Helped |
| Use TLM instead of other backward-policy approaches | MDQN · Bit Sequences | number of discovered modes | No clear effect |
| Use TLM instead of uniform and naive backward policies | TB and SubTB · Bit Sequences | number of discovered modes | No clear effect |
| Use TLM instead of other backward-policy approaches | DB and SubTB · Bit Sequences | Spearman correlation | Helped |
| Use TLM instead of other backward-policy approaches | DB · QM9 | number of discovered modes | Helped |
| Use TLM instead of other backward-policy approaches | MDQN and SubTB · QM9 | number of discovered modes | No clear effect |
| Use TLM instead of other backward-policy approaches | GFlowNet algorithms · sEH | number of discovered modes | Mixed |
| Use TLM instead of other backward-policy approaches | SubTB · QM9 | Pearson correlation | Helped |
Surprising findings
- On sEH, the uniform fixed backward policy often matches or beats learned backward policies, which the authors attribute to the structured fragment-based environment.
- All methods fail to converge on the hard hypergrid reward design with TB.
Limitations
- On sEH, TLM can underperform other backward approaches in mode discovery.
- Convergence guarantee relies on stability and no-regret assumptions that are conjectured rather than proven for the practical algorithms.
- Results are reported over three seeds on relatively small benchmarks.
Experimental setup
Datasets: QM9 sEH proxy dataset
Benchmarks: hypergrid bit sequences sEH QM9
Metrics: L1 distance to target distribution number of discovered modes Spearman correlation Pearson correlation
Baselines: uniform backward policy naive backward optimization maximum entropy backward policy pessimistic backward policy
Builds on: trajectory balance subtrajectory balance soft DQN munchausen DQN pessimistic backward policy
Closest papers by meaning
- GFlowNet Training by Policy Gradients (ICML 2024 · similarity 0.94)
- Looking Backward: Retrospective Backward Synthesis for Goal-Conditioned GFlowNets (ICLR 2025 · similarity 0.94)
- Pessimistic Backward Policy for GFlowNets (NeurIPS 2024 · similarity 0.93)
- Random Policy Evaluation Uncovers Policies of Generative Flow Networks (ICML 2025 · similarity 0.93)
- Pre-Training and Fine-Tuning Generative Flow Networks (ICLR 2024 · similarity 0.92)
- Flow Factorization for Efficient Generative Flow Networks (AAAI 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.