Research map / Probabilistic and scientific modeling / Variational and generative sampling
Joint Learning of Energy-based Models and their Partition Function
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a min-min formulation that jointly learns an energy function and its log-partition (parameterized as a neural network) to train probabilistic EBMs in combinatorial discrete spaces via MCMC-free doubly stochastic SGD, extended to Fenchel-Young losses including sparsemax.
Problem
Maximum likelihood learning of energy-based models over combinatorially-large discrete output spaces (sets, permutations) is intractable because the partition function cannot be computed, and MCMC-based estimates are biased and costly.
Approach
Introduce the log-partition as a separate function minimized jointly with the energy (a Lagrange multiplier via duality), so the objective is an expectation over prior samples without MCMC; parameterize the log-partition with a neural network that generalizes to unseen inputs, and extend to Fenchel-Young losses (e.g., sparsemax via chi-square f-divergence).
Key results
- On multilabel classification, the min-min logistic loss with MLP/ResNet energy models reaches f1 around 65 on yeast and 75 on scene, outperforming the MCMC logistic baseline on several datasets (e.g., cal500 45.0 vs 44.6 for unary models).
- On label ranking with the Birkhoff polytope and MLP energy, min-min logistic achieves Kendall tau 90.80 on authorship, versus 89.68 for MCMC.
- The learned log-partition on the unary model correlates strongly with the exact log-sum-exp on test samples, and the approach converges to exact MLE as the number of prior samples grows.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use logistic min-min instead of the strongest reported MCMC baseline | Multilabel classification models · Yeast | F1: 61.98 -> 63.41 | Helped |
| Use sparsemax min-min instead of the strongest reported MCMC baseline | Multilabel classification models · Scene | F1: 71.92 -> 75.83 | Helped |
| Use logistic min-min instead of the strongest reported MCMC baseline | Multilabel classification models · Birds | F1: 44.93 -> 46.48 | Helped |
| Use logistic min-min instead of the strongest reported MCMC baseline | Multilabel classification models · Emotions | F1: 63.15 -> 67.45 | Helped |
| Use sparsemax min-min instead of the strongest reported non-min-min baseline | Multilabel classification models · Cal500 | F1: 47.67 -> 46.39 | Hurt |
| Use logistic min-min instead of the strongest reported MCMC baseline | Label ranking models · Authorship | Kendall rank correlation: 89.68 -> 90.80 | Helped |
| Use sparsemax min-min instead of the strongest reported MCMC baseline | Label ranking models · Glass | Kendall rank correlation: 80.67 -> 85.94 | Helped |
| Use sparsemax min-min instead of the strongest reported MCMC baseline | Label ranking models · Iris | Kendall rank correlation: 97.78 -> 99.26 | Helped |
| Use sparsemax min-min instead of the strongest reported MCMC baseline | Label ranking models · Vehicle | Kendall rank correlation: 86.34 -> 87.52 | Helped |
| Use logistic min-min instead of the strongest reported MCMC baseline | Label ranking models · Vowel | Kendall rank correlation: 57.90 -> 74.26 | Helped |
| Use logistic min-min instead of the strongest reported MCMC baseline | Label ranking models · Wine | Kendall rank correlation: 96.91 -> 95.06 | Hurt |
| Use fewer prior samples for the min-min objective (ablation) | Proposed logistic min-min method · Cal500 | F1 | Helped |
Surprising findings
- Using fewer prior samples (y') can give better test f1 than exact MLE, suggesting a regularization effect.
- The min-max adversarial baseline performs poorly on several datasets, e.g., scene with linear unary gets 52.29 f1.
Limitations
- Evaluated primarily on prediction (mode finding), not on sampling from p(y|x).
- Only neural-network approximation of tau gives approximate MLE; exact equivalence holds only in function space.
- Experiments are on small benchmark datasets with a unary/pairwise model family.
Experimental setup
Datasets: yeast scene birds emotions cal500 mediamill authorship glass iris vehicle vowel wine
Benchmarks: yeast scene birds emotions cal500 authorship glass iris vehicle vowel wine
Metrics: f1-score kendall rank correlation
Baselines: MCMC maximum likelihood (Metropolis-Hastings) min-max adversarial EBM training (REINFORCE) generalized Fenchel-Young loss exact MLE on unary models
Builds on: Fenchel-Young losses conditional random fields generalized perceptron loss input-convex neural networks
Closest papers by meaning
- No MCMC for me: Amortized sampling for fast and stable training of energy-based models (ICLR 2021 · similarity 0.90)
- Energy-Based Modelling for Discrete and Mixed Data via Heat Equations on Structured Spaces (NeurIPS 2024 · similarity 0.89)
- On Energy-Based Models with Overparametrized Shallow Neural Networks (ICML 2021 · similarity 0.89)
- Energy-Inspired Models: Learning with Sampler-Induced Distributions (NeurIPS 2019 · similarity 0.88)
- Learning Latent Space Hierarchical EBM Diffusion Models (ICML 2024 · similarity 0.86)
- Learning Energy-Based Models by Cooperative Diffusion Recovery Likelihood (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.