Research map / Probabilistic and scientific modeling / Variational and generative sampling
Energy-Based Modelling for Discrete and Mixed Data via Heat Equations on Structured Spaces
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains discrete and mixed-state energy-based models with energy discrepancy, using CTMC heat-equation perturbations on graph-structured spaces (uniform, cyclical, ordinal) to avoid MCMC.
Problem
Training energy-based models on discrete or mixed (tabular) state spaces is hard because contrastive divergence needs MCMC sampling that mixes poorly and yields biased energies.
Approach
Extends energy discrepancy (a sampling-free contrastive loss) to discrete spaces by defining perturbations as continuous-time Markov chains whose rate matrix is a graph Laplacian (uniform, cyclical, ordinal, masking), and combines Gaussian perturbations for numerical features to handle mixed data.
Key results
- On binary image modelling, ED-Grid achieves 93.94 NLL on Omniglot vs 94.72 for GWG, the best baseline shown there; on MNIST, GWG (80.01) remains better than ED-Grid (90.61).
- On real tabular datasets, TabED-Uni attains average rank 3.83 versus CTGAN 6.33 and TVAE 5.17, while TabDDPM is best at 1.5.
- Theory: energy discrepancy converges to maximum likelihood loss at an exponential rate governed by the spectral gap of the rate matrix, with bound sqrt(S) exp(-|lambda_2| t) KL.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ED-Bern instead of EB-GFN | discrete density estimation · NLL across 2spirals, circles, moons, pinwheel, swissroll, and checkerboard | NLL: 20.050 -> 20.039; 20.546 -> 20.601; 19.732 -> 19.710; 19.554 -> 19.568; 20.146 -> 20.084; 20.696 -> 20.679 | Mixed |
| Use ED-Grid instead of the strongest shown baseline | discrete density estimation · MMD across 2spirals, 8gaussians, circles, and checkerboard | MMD: 0.149 -> 0.097; 0.078 -> -0.066; 0.188 -> 0.022; 1.206 -> 2.049 | Helped |
| Use ED-Bern instead of the strongest shown baseline | discrete density estimation · MMD on moons, pinwheel, and swissroll | MMD: 0.121 -> -0.088; 1.746 -> 0.046; 1.382 -> 0.045 | Helped |
| Use ED-Grid instead of GWG | discrete image modelling · Static MNIST and Dynamic MNIST | NLL: 80.01 -> 90.61; 80.51 -> 90.19 | Hurt |
| Use ED-Grid instead of GWG | discrete image modelling · Omniglot | NLL: 94.72 -> 93.94 | Helped |
| Use ED training rather than the conditional-likelihood classifier | Adult classifier · Adult | expected calibration error | Helped |
| Use ED-Bern rather than ED-Grid (ablation) | binary discrete density estimation · NLL and MMD across synthetic datasets | NLL and MMD | Mixed |
| Use uniform categorical perturbation rather than structured perturbations (ablation) | TabED · real-world tabular datasets | average rank: 3.83 vs 4.83 | Helped |
Surprising findings
- The unstructured uniform perturbation (TabED-Uni) performs slightly better than structured cyclical/ordinal variants on the real tabular datasets.
- Contrastive divergence performs poorly on mixed-state tabular data.
Limitations
- Sensitive to the assumption that the data distribution is positive on the whole state space; manifold hypothesis violated on binary images.
- Does not match contrastive divergence with many MCMC steps on binary image data.
- Only uniform, cyclical, and ordinal (and masking) graph structures considered.
Experimental setup
Datasets: 2D synthetic distributions (2spirals, 8gaussians, circles, moons, pinwheel, swissroll, checkerboard) converted to discre Adult Bank Cardio Churn Mushroom Beijing Static MNIST Dynamic MNIST Omniglot synthetic ring mixed-type dataset
Benchmarks: Adult Bank Cardio Churn Mushroom Beijing Static MNIST Dynamic MNIST Omniglot 2D discrete density datasets
Metrics: AUC RMSE NLL MMD expected calibration error average rank
Baselines: PCD ALOE+ EB-GFN Gibbs GWG DULA CTGAN TVAE TabDDPM TabCD
Builds on: energy discrepancy contrastive divergence discrete diffusion models concrete score matching TabDDPM
Closest papers by meaning
- No MCMC for me: Amortized sampling for fast and stable training of energy-based models (ICLR 2021 · similarity 0.89)
- Joint Learning of Energy-based Models and their Partition Function (ICML 2025 · similarity 0.89)
- Learning Energy-Based Models by Cooperative Diffusion Recovery Likelihood (ICLR 2024 · similarity 0.86)
- A Theoretical Framework For Overfitting In Energy-based Modeling (ICML 2025 · similarity 0.86)
- TabEBM: A Tabular Data Augmentation Method with Distinct Class-Specific Energy-Based Models (NeurIPS 2024 · similarity 0.86)
- Scalable Discrete Diffusion Samplers: Combinatorial Optimization and Statistical Physics (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.