Research map / Probabilistic and scientific modeling / Variational and generative sampling
On Divergence Measures for Training GFlowNets
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Revisits GFlowNet training as hierarchical variational inference and trains GFlowNets by minimizing Renyi-alpha, Tsallis-alpha, forward and reverse KL divergences with REINFORCE leave-one-out and autodiff-friendly control variates to reduce gradient variance.
Problem
Training GFlowNets usually minimizes trajectory/detailed balance log-squared losses; directly minimizing KL-type divergences was reported as biased and high-variance, so their potential as learning objectives was unclear.
Approach
Derive gradient estimators for Renyi-alpha, Tsallis-alpha, forward and reverse KL objectives for GFlowNets, and reduce their variance with score-function control variates, a batch-estimated optimal baseline, and a REINFORCE leave-one-out estimator, showing these divergences often converge faster than TB loss. It also extends the TB-KL gradient equivalence to arbitrary topological spaces.
Key results
- Divergence-based objectives (reverse KL, forward KL, Renyi-alpha, Tsallis-alpha) often match or beat the TB loss in convergence speed across tasks, with alpha=0.5 for Renyi and Tsallis.
- On the mixture-of-Gaussians task, forward KL reached 0.09 +/- 0.10 vs 0.31 +/- 0.08 for TB, while on sets reverse KL, forward KL, Renyi and Tsallis all reached 0.03 vs 0.07 for TB.
- Control variates (REINFORCE baseline and leave-one-out) substantially reduce gradient variance and stabilize training, especially for reverse KL.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Minimize reverse KL instead of TB loss | GFlowNet · Sequences | L1 distance: 0.28 ± 0.06 -> 0.16 ± 0.06 | Helped |
| Minimize forward KL instead of TB loss | GFlowNet · GMs | Jensen-Shannon divergence: 0.31 ± 0.08 -> 0.09 ± 0.10 | Helped |
| Minimize Renyi-α divergence (α = 0.5) instead of TB loss | GFlowNet · Sets | L1 distance: 0.07 ± 0.00 -> 0.03 ± 0.00 | Helped |
| Minimize Tsallis-α divergence (α = 0.5) instead of TB loss | GFlowNet · Sets | L1 distance: 0.07 ± 0.00 -> 0.03 ± 0.00 | Helped |
| Use divergence-based objectives instead of TB loss | GFlowNet · BPI | L1 distance: null | No clear effect |
| Add control variates to the REINFORCE gradient estimator | Divergence-trained GFlowNet · Generative tasks | training convergence speed | Helped |
Limitations
- Results are only for alpha=0.5 for the Renyi and Tsallis objectives; other alpha values not thoroughly explored.
- Forward KL estimator relies on importance sampling with high variance.
- Experiments are on small synthetic benchmark tasks.
Experimental setup
Datasets: synthetic set generation tasks synthetic autoregressive sequence generation Bayesian phylogenetic inference datasets mixture of Gaussians target banana-shaped distribution
Benchmarks: set generation autoregressive sequence generation Bayesian phylogenetic inference mixture of Gaussians banana-shaped distribution
Metrics: L1 distance between learned and target distribution Jensen-Shannon divergence gradient variance (trace of covariance) convergence speed
Baselines: Trajectory balance loss Detailed balance Reverse KL Forward KL Renyi-alpha Tsallis-alpha
Builds on: GFlowNets trajectory balance hierarchical variational inference for GFlowNets REINFORCE leave-one-out Renyi-alpha divergence Tsallis-alpha divergence
Closest papers by meaning
- Beyond Squared Error: Exploring Loss Design for Enhanced Training of Generative Flow Networks (ICLR 2025 · similarity 0.93)
- Pre-Training and Fine-Tuning Generative Flow Networks (ICLR 2024 · similarity 0.91)
- Looking Backward: Retrospective Backward Synthesis for Goal-Conditioned GFlowNets (ICLR 2025 · similarity 0.90)
- GFlowNet Training by Policy Gradients (ICML 2024 · similarity 0.90)
- Flow Factorization for Efficient Generative Flow Networks (AAAI 2025 · similarity 0.90)
- Revisiting Non-Acyclic GFlowNets in Discrete Environments (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.