Research map / Probabilistic and scientific modeling / Variational and generative sampling
REMEDI: Corrective Transformations for Improved Neural Entropy Estimation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes REMEDI, which estimates differential entropy by fitting a KNIFE Gaussian-mixture base density by cross-entropy and then correcting it with a Donsker-Varadhan neural term, and applies it to entropy estimation, Information Bottleneck, and sampling.
Problem
Accurate differential entropy estimation becomes data-inefficient in moderately high dimensions, and existing plug-in estimators (KDE, KNIFE) overfit or fail to approach the true entropy; mutual information estimation in the Information Bottleneck inherits this problem.
Approach
Combine a simple learnable base density q (a KNIFE Gaussian mixture trained by cross-entropy) with a neural correction T fitted via the Donsker-Varadhan bound, so that H(P) is upper-bounded by cross-entropy minus the DV estimate of relative entropy; the corrected density q*exp(T) then approximates p, and the paper proves consistency under sub-Gaussian (non-compact) assumptions.
Key results
- On 8-dimensional triangle mixture (true entropy 2.585), REMEDI gives 3.08 vs KNIFE 4.36.
- On two moons, REMEDI pushes entropy estimate below 0.30 from KNIFE's ~0.45, close to oracle ~0.29.
- In IB experiments REMEDI gives best accuracy on MNIST (98.36%) and CIFAR-10 (86.99% with KNIFE base), and matches VIB on ImageNet (79.81% vs 79.83%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add REMEDI correction to KNIFE | KNIFE entropy estimator · 8-dimensional triangle | entropy estimate: 4.3563 ± 0.0528 -> 3.0798 ± 0.0368 | Helped |
| Use Gaussian instead of KNIFE as the REMEDI base (ablation) | REMEDI · MNIST test | accuracy (%): 98.28 (0.105) -> 98.36 (0.076) | Helped |
| Use Gaussian instead of KNIFE as the REMEDI base (ablation) | REMEDI · CIFAR-10 test | accuracy (%): 86.99 (0.176) -> 85.74 (1.671) | Hurt |
| Use Gaussian instead of KNIFE as the REMEDI base (ablation) | REMEDI · ImageNet test | accuracy (%): 79.61 (0.015) -> 79.81 (0.047) | Helped |
| Use REMEDI (Gaussian base) instead of VIB | Information Bottleneck · MNIST test | accuracy (%): 98.34 (0.070) -> 98.36 (0.076) | Helped |
| Use REMEDI (Gaussian base) instead of VIB | Information Bottleneck · CIFAR-10 test | accuracy (%): 86.45 (0.135) -> 85.74 (1.671) | Hurt |
| Use REMEDI (KNIFE base) instead of VIB | Information Bottleneck · CIFAR-10 test | accuracy (%): 86.45 (0.135) -> 86.99 (0.176) | Helped |
| Use REMEDI (Gaussian base) instead of VIB | Information Bottleneck · ImageNet test | accuracy (%): 79.83 (0.061) -> 79.81 (0.047) | No clear effect |
Surprising findings
- A simple isotropic Gaussian base distribution gave better IB classification accuracy than the trainable KNIFE base on MNIST and ImageNet.
- MINE showed stability issues on CIFAR-10 and ImageNet, attributed to the exponential term in the Donsker-Varadhan bound.
Limitations
- Authors note the method has not been tested in very high dimensions such as raw MNIST image entropy estimation.
- Consistency proof requires sub-Gaussian base distribution and compact parameter space, and needs large batch sizes to reduce bias in the DV log-expectation gradient.
- Gains on ImageNet over VIB are not clear and the MI estimate is lower than VIB's, so accuracy is roughly tied.
Experimental setup
Datasets: two moons (scikit-learn) triangle mixture (synthetic) uniform ball (synthetic) MNIST CIFAR-10 ImageNet
Benchmarks: two moons entropy 8-dimensional triangle entropy MNIST IB classification CIFAR-10 IB classification ImageNet IB classification
Metrics: entropy estimate test classification error test accuracy log-likelihood mutual information estimate (bits)
Baselines: KNIFE MINE VIB
Builds on: KNIFE Donsker-Varadhan representation Information Bottleneck MINE VIB
Closest papers by meaning
- A Unifying Information-theoretic Perspective on Evaluating Generative Models (AAAI 2025 · similarity 0.85)
- MINDE: Mutual Information Neural Diffusion Estimation (ICLR 2024 · similarity 0.84)
- Provably Reliable Classifier Guidance via Cross-Entropy Control (NeurIPS 2026 · similarity 0.83)
- Mutual Information Estimation via $f$-Divergence and Data Derangements (NeurIPS 2024 · similarity 0.83)
- EVODiff: Entropy-aware Variance Optimized Diffusion Inference (NeurIPS 2025 · similarity 0.83)
- Generative Cross-Entropy: A Strictly Proper Loss for Data-Efficient Classification (NeurIPS 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.