Research map / Probabilistic and scientific modeling / Variational and generative sampling
A Differential Entropy Estimator for Training Neural Networks
Georg Pichler, Pierre Jean A. Colombo, Malik Boudiaf, Günther Koliander, Pablo Piantanida
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use KNIFE-based MI estimation instead of competing MI estimators or VIBERT in the information bottleneck objective | BERT fine-tuning · GLUE: MRPC, STS-B, RTE | task scores: MRPC F1 85.1 -> 85.3; MRPC accuracy 89.1 -> 90.1; STS-B Pearson 90.0 -> 90.3; STS-B Spearman 89.7 -> 90.1; RTE accuracy | Improved |
| Use KNIFE-based MI estimation rather than the competing estimators | Unsupervised domain adaptation · Mean across 6 source/target tasks | accuracy: 82.8 -> 83.4 | Improved |
| Use KNIFE-based MI estimation rather than other MI estimators | Fair textual classification · DIAL sentiment and mention labels | main-task accuracy at perfect privacy | Mixed |
| Allow kernel means and mixture weights to be learned, rather than fixing kernel support | Differential entropy estimator · Changing input distribution | adaptation to distribution shift | Improved |
| Use a flexible kernel mixture rather than a single Gaussian density | Differential entropy estimator · Multimodal triangle-mixture distributions | density matching | Improved |
Closest papers by meaning
- InfoNet: Neural Estimation of Mutual Information without Test-Time Optimization (ICML 2024 · similarity 0.88)
- A Benchmark Suite for Evaluating Neural Mutual Information Estimators on Unstructured Datasets (NeurIPS 2024 · similarity 0.87)
- Mutual Information Estimation via $f$-Divergence and Data Derangements (NeurIPS 2024 · similarity 0.87)
- REMEDI: Corrective Transformations for Improved Neural Entropy Estimation (ICML 2024 · similarity 0.87)
- MINDE: Mutual Information Neural Diffusion Estimation (ICLR 2024 · similarity 0.86)
- Connecting Jensen–Shannon and Kullback–Leibler Divergences: A New Bound for Representation Learning (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
