Research map / Image generation / Fast diffusion sampling
Neural Entropy
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Defines "neural entropy" as the information a diffusion model stores, estimated via an entropy-matching loss tied to total entropy production, and measures it on toy Gaussian-mixture MLP diffusion models and MNIST U-Net models.
Problem
How much information (in bytes/nats) does a trained diffusion model store about its data distribution, given that diffusion erases information and the network must reinstate it?
Approach
Connects diffusion models to stochastic thermodynamics: the total entropy produced by the forward diffusion lower-bounds the KL between data and generated distributions, so the information a perfectly trained network stores equals total entropy; a practical estimator neural entropy S_NN = integral of (g^2/2)E||e_theta||^2 is proposed from an entropy-matching objective.
Key results
- On the Gaussian-mixture VPx and SL models, the learned S_NN approaches the ideal total entropy S_tot as training epochs increase, and the VP model (lower entropy) trains faster and reconstructs better than VPx or SL.
- For an unconditional MNIST VP U-Net, neural entropy and cross-entropy saturate during training, and S_NN(T) grows roughly logarithmically with the number of training samples N.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use regular VP instead of VPx or SL | MLP diffusion model · 5-component Gaussian mixture, 6D | reconstruction accuracy | Helped |
| use VPx or SL with weak noise and relatively strong drift instead of regular VP | MLP diffusion model · 5-component Gaussian mixture, 6D | total entropy | Hurt |
| increase the number of training samples | MNIST unconditional VP diffusion model · MNIST | neural entropy | Helped |
| increase the number of training samples | MNIST unconditional VP diffusion model · MNIST | neural entropy scaling with sample count | Mixed |
| train for more epochs | MLP diffusion model · 5-component Gaussian mixture, 6D | neural entropy | Helped |
| continue training | MNIST unconditional VP diffusion model · MNIST | neural entropy | No clear effect |
Surprising findings
- Setting the score-matching network to zero gives zero total entropy for a time-independent process with data equal to the equilibrium distribution, yet the score is nonzero, so score-matching loss cannot be interpreted directly as an entropy.
- Faster forward diffusion (e.g., straight-line trajectories) produces more total entropy, so a diffusion model must store more information to reverse it.
Limitations
- Neural entropy is defined only for continuous diffusion models; score-matching models are not handled cleanly.
- The logarithmic growth of neural entropy with dataset size is an empirical observation without theoretical explanation.
- Entropy production diverges near s=0 for image models, requiring care in computation.
Experimental setup
Datasets: MNIST synthetic Gaussian mixture (5 components, 6D)
Benchmarks: MNIST synthetic Gaussian mixture
Metrics: neural entropy S_NN entropy production rate cross-entropy KL divergence training loss
Baselines: Straight Line Diffusion Model (SL) VP diffusion score-matching diffusion
Builds on: score-based diffusion models flow matching Jarzynski equality Straight Line Diffusion Model Variance Preserving SDE
Closest papers by meaning
- MINDE: Mutual Information Neural Diffusion Estimation (ICLR 2024 · similarity 0.86)
- Conservation Laws for Diffusion Models (NeurIPS 2026 · similarity 0.86)
- Shallow diffusion networks provably learn hidden low-dimensional structure (ICLR 2025 · similarity 0.85)
- When Diffusion Models Memorize: Inductive Biases in Probability Flow of Minimum-Norm Shallow Neural Nets (ICML 2025 · similarity 0.85)
- Understanding Generalizability of Diffusion Models Requires Rethinking the Hidden Gaussian Structure (NeurIPS 2024 · similarity 0.85)
- Feature Information Dynamics in Diffusion (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.