Research map / Probabilistic and scientific modeling / Variational and generative sampling
Expected flow networks in stochastic environments and two-player zero-sum games
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Extends GFlowNets to stochastic environments via expected flow networks (EFlowNets) and to two-player zero-sum games via adversarial flow networks (AFlowNets), trained by self-play with a branch-adjusted trajectory-balance loss.
Problem
GFlowNets assume deterministic transitions, and prior stochastic GFlowNet formulations can be unsatisfiable and sensitive to the training policy; sequential sampling in adversarial games also lacks a principled GFlowNet treatment.
Approach
Replace the fixed environment-as-action treatment with expected detailed balance constraints that integrate over environment transitions (EFlowNets), then model each player in a game as an EFlowNet agent facing the other players' policies as environment (AFlowNets), with a trajectory-balance loss for branch-adjusted zero-sum rewards.
Key results
- In Connect-4, AFlowNet with lambda 10 or 15 reaches an Elo roughly 800 points above AlphaZero, winning almost every game against AlphaZero.
- AFlowNets learn to play optimal moves in >80% of Connect-4 board states after 3 hours on one RTX8000 GPU.
- On the stochastic protein design task, EFlowNets outperform Stoch-GFN, GFN, and SAC in diversity and top-100 reward when rewards are peaky (large beta).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use EFlowNet expected-flow objectives instead of stochastic GFlowNet constraints | GFlowNet protein-design sampler · TFBind stochastic protein design | diverse modes found | Helped |
| Use EFlowNet expected-flow objectives instead of stochastic GFlowNet constraints | GFlowNet protein-design sampler · TFBind stochastic protein design | top-100 mean reward | Helped |
| Increase reward exponent β (ablation) | EFlowNet and other protein-design samplers · TFBind stochastic protein design | top-100 mean reward | Mixed |
| Set λ=10 rather than λ=2 or λ=15 (ablation) | EDB-trained AFlowNet · Tic-tac-toe | Elo: 357.2±12.7; 363.2±7.7; 356.9±11.6 | Mixed |
| Increase λ from 2 to 10 to 15 (ablation) | AFlowNet · Connect-4 | Elo: 1190.8 ± 64.2; 1700.1 ± 60.0; 1835.3 ± 154.9 | Helped |
| Use trajectory-balance training instead of EDB training (ablation) | AFlowNet_10 · Tic-tac-toe | training convergence speed: about 10 times faster | Helped |
| Use trajectory-balance training instead of EDB training (ablation) | AFlowNet · Tic-tac-toe | Elo | Hurt |
| Vary λ in TB-trained AFlowNet (ablation) | TB-trained AFlowNet · Tic-tac-toe | Elo | No clear effect |
| Apply branch-factor adjustment to terminal rewards (ablation) | AFlowNet for complex games · Two-player games | game-playing performance | Helped |
Surprising findings
- Tic-tac-toe AFlowNets require roughly an order of magnitude more optimization steps than AlphaZero with MCTS to reach similar Elo.
- The AFlowNet does not need to learn exact flows to achieve strong game-playing performance.
Limitations
- Trajectory-level TB objective suffers high variance and memory cost for long trajectories.
- Scaling to larger games such as chess and Go is untested; tic-tac-toe AFlowNets converge slower than AlphaZero.
Experimental setup
Datasets: TFBind protein-design environment (DNA 8-mer binding proxy) tic-tac-toe self-play Connect-4 self-play
Benchmarks: TFBind protein design (top-100 reward, number of modes) tic-tac-toe tournament and Elo Connect-4 tournament and Elo Connect-4 perfect-solver optimal move rate
Metrics: top-100 mean reward number of modes (sphere exclusion) Elo (BayesElo) tournament win/draw/loss score percent optimal moves
Baselines: Stoch-GFN GFN discrete soft actor-critic AlphaZero AlphaZero+MCTS DQN SoftDQN tree search agent uniform random agent
Builds on: GFlowNet trajectory balance detailed balance AlphaZero stochastic GFlowNet
Closest papers by meaning
- Flow Factorization for Efficient Generative Flow Networks (AAAI 2025 · similarity 0.92)
- Pre-Training and Fine-Tuning Generative Flow Networks (ICLR 2024 · similarity 0.90)
- A Theory of Non-acyclic Generative Flow Networks (AAAI 2024 · similarity 0.89)
- Order-Preserving GFlowNets (ICLR 2024 · similarity 0.89)
- Revisiting Non-Acyclic GFlowNets in Discrete Environments (ICML 2025 · similarity 0.89)
- Pessimistic Backward Policy for GFlowNets (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.