Research map / Probabilistic and scientific modeling / Variational and generative sampling
Discrete Neural Flow Samplers with Locally Equivariant Transformer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Discrete Neural Flow Samplers (DNFS), which learn a one-way CTMC rate matrix via the Kolmogorov equation with control-variate-based coordinate descent and a locally equivariant Transformer (leTF) parameterisation for sampling discrete distributions.
Problem
Sampling from unnormalised discrete distributions is hard because MCMC mixes slowly, and learning a neural sampler requires handling an intractable partition-function time derivative and expensive per-neighbour network evaluations.
Approach
Learn a CTMC rate matrix satisfying the Kolmogorov forward equation, estimate the log-partition derivative with a discrete Stein control variate (coordinate descent), and parameterise the rate matrix as a one-way matrix from a locally equivariant Transformer so one forward pass gives all neighbour rates.
Key results
- On MIS over ER64-75, DNFS reaches an 11.8% drop from Gurobi, versus 6.91% for GFlowNet and 4.91% for DMALA.
- DNFS+DMALA (MCMC refinement after DNFS) reduces the drop to 0.11% on ER16-20 and 2.38% on ER64-75.
- On the Ising model, DNFS performs competitively with LEAPS and outperforms GFlowNet in capturing the energy distribution.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add DMALA refinement steps to DNFS | DNFS on Erdős–Rényi MIS graphs · ER16-20, ER32-40, ER64-75 | solution size: 8.28 -> 8.91; 13.18 -> 14.31; 18.12 -> 20.06 | Helped |
| Train DNFS rather than use random initialization | DNFS on Erdős–Rényi MIS graphs · ER16-20, ER32-40, ER64-75 | solution size: 5.21 -> 8.28; 6.31 -> 13.18; 8.63 -> 18.12 | Helped |
| Use DNFS instead of GFlowNet | Neural samplers for Erdős–Rényi MIS · ER16-20, ER32-40, ER64-75 | solution size: 8.75 -> 8.28; 13.93 -> 13.18; 19.13 -> 18.12 | Hurt |
Surprising findings
- Control-variate estimator of the log-partition derivative yields lower variance and lower training loss than direct Monte Carlo estimation.
- Standalone DNFS trails the GFlowNet and MCMC baselines on MIS, with the MCMC refinement needed to reach strong performance.
Limitations
- Extension beyond binary state spaces is hard due to the cost of computing probability ratios; a naive Taylor approximation is biased.
- Struggles to scale to very high-dimensional distributions, with loss values that can explode.
- Standalone performance on MIS lags GFlowNet and MCMC baselines.
Experimental setup
Datasets: Ising model (10x10 lattice) 2D synthetic distributions Gray-coded to 32-bit Erdos-Renyi random graphs for MIS
Benchmarks: Ising model sampling (10x10, sigma=0.1) pre-trained EBM sampling probability mass estimation ER16-20/ER32-40/ER64-75 maximum independent set
Metrics: effective sample size log RMSE energy histogram solution size approximation ratio inference time
Baselines: Gibbs sampling GFlowNet LEAPS DMALA PCD ED-GFN
Builds on: LEAPS discrete flow models hollow transformer PINN loss
Closest papers by meaning
- Scalable Discrete Diffusion Samplers: Combinatorial Optimization and Statistical Physics (ICLR 2025 · similarity 0.89)
- MDNS: Masked Diffusion Neural Sampler via Stochastic Optimal Control (NeurIPS 2025 · similarity 0.89)
- LEAPS: A discrete neural sampler via locally equivariant networks (ICML 2025 · similarity 0.89)
- Diffusion Generative Flow Samplers: Improving learning signals through partial trajectory optimization (ICLR 2024 · similarity 0.88)
- Ai-sampler: Adversarial Learning of Markov kernels with involutive maps (ICML 2024 · similarity 0.87)
- Annealing Flow Generative Models Towards Sampling High-Dimensional and Multi-Modal Distributions (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.