Research map / Architectures and efficiency / RNNs and normalization
Information Bottleneck Analysis of Deep Neural Networks via Lossy Compression
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an information-bottleneck analysis framework for general DNNs that estimates mutual information between compressed (PCA/autoencoder latent) representations of high-dimensional stochastic network layers, validated on synthetic data and a small MNIST CNN.
Problem
Mutual information between high-dimensional hidden layer outputs and inputs/targets is hard to estimate (sample complexity exponential in dimension and MI infinite for deterministic nets), so the fitting-compression hypothesis of the information bottleneck theory has only been tested on tiny network
Approach
Combine stochastic (Gaussian-dropout) network layers with lossy compression of inputs and layer outputs (autoencoder for inputs, PCA for layers) before applying a weighted Kozachenko-Leonenko entropy estimator, with theoretical bounds on the MI error introduced by compression.
Key results
- On 16x16 and 32x32 synthetic images, the weighted Kozachenko-Leonenko estimator on PCA/AE-compressed data gives the most accurate MI estimates among the tested methods, outperforming MINE and classical estimators.
- For 4x4 and 8x8 image tests, uncompressed classical estimators have MSE from 9.7 to 4.1e3 nats, while WKL with PCA compression reaches 1.87 and 1.71 nats.
- On the MNIST CNN, the information plane shows no clear large-scale compression phase; the number of fitting and compression phases varies by layer, and the first compression coincides with accelerated loss decrease.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply PCA compression before WKL | WKL MI estimation · 4 × 4 images of 2D Gaussians | MSE (nats): 2,9 \cdot 10^1 -> 1,87 | Helped |
| Apply PCA compression before WKL | WKL MI estimation · 8 × 8 images of 2D Gaussians | MSE (nats): 7,5 \cdot 10^1 -> 1,71 | Helped |
| Use PCA compression before WKL instead of uncompressed estimation | WKL MI estimation · 16 × 16 images of 2D Gaussians | MSE (nats): -- -> 0,67 | Helped |
Surprising findings
- Classical entropy estimators (KDE, KL, WKL) fail on uncompressed high-dimensional images, with MSE in the thousands of nats for 8x8 images.
- No clear large-scale compression phase is observed in the MNIST information plane, casting doubt on the original fitting-compression hypothesis.
Limitations
- Compression of arbitrary amounts of information is possible; bounds require additional assumptions such as linear compression and smoothed distributions.
- Lossy compression with PCA is used for layers, and the authors note the approach relies on manifold hypothesis and stochastic proxies rather than the deterministic original network.
- Experiments are restricted to a small MNIST CNN, so conclusions about large networks remain untested.
Experimental setup
Datasets: MNIST synthetic rasterized 2D Gaussian images synthetic rasterized rectangle images
Benchmarks: synthetic MI benchmarks with predefined mutual information MNIST information plane
Metrics: estimated MI vs true MI MSE in nats
Baselines: MINE KDE maximum-likelihood KDE least squares cross-validation non-weighted Kozachenko-Leonenko
Builds on: information bottleneck MINE stochastic neural networks Kozachenko-Leonenko estimator Shwartz-Ziv & Tishby fitting-compression hypothesis
Closest papers by meaning
- Information Bottleneck: Exact Analysis of (Quantized) Neural Networks (ICLR 2022 · similarity 0.95)
- On the Information Bottleneck Theory of Deep Learning (ICLR 2018 · similarity 0.92)
- Cauchy-Schwarz Divergence Information Bottleneck for Regression (ICLR 2024 · similarity 0.91)
- Structured IB: Improving Information Bottleneck with Structured Feature Learning (AAAI 2025 · similarity 0.89)
- Understanding and Leveraging the Learning Phases of Neural Networks (AAAI 2024 · similarity 0.89)
- A Rate-Distortion View of Uncertainty Quantification (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.