Research map / Learning theory and algorithms / Kernels and regression theory
How Does Information Bottleneck Help Deep Learning?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| combine model compression with conditional representation compression | 2D clustered classification models · 2D clustered classification | Pearson correlation with loss generalization gap: 0.3842 -> 0.4130 | Helped |
| combine model compression with representation compression | MNIST models · MNIST | Pearson correlation with loss generalization gap: 0.5467 -> 0.7626 | Helped |
| combine model compression with representation compression | Fashion MNIST models · Fashion MNIST | Pearson correlation with loss generalization gap: 0.5146 -> 0.6488 | Helped |
| take the minimum across layers rather than the mean (ablation) | CIFAR10 model + conditional representation compression metric · CIFAR10 | Spearman correlation with loss generalization gap: 0.4429 -> 0.8632 | Helped |
| take the minimum across layers rather than the maximum (ablation) | CIFAR10 model + conditional representation compression metric · CIFAR10 | Spearman correlation with loss generalization gap: 0.5711 -> 0.8632 | Helped |
| add information bottleneck feature regularization (ablation) | feature-compression metrics in binned 2D clustering models · 2D clustered classification with feature binning | Pearson correlation with loss generalization gap: 0.3032 -> -0.0440 | Hurt |
| estimate mutual information using a sample 10 times larger than the training set (ablation) | representation-compression metrics · 2D clustered classification | predictive ability for generalization gap | Helped |
| use MLE rather than adaptive KDE to select noise variance (ablation) | CIFAR10 mutual-information estimates · CIFAR10 | predictive ability for generalization gap | Helped |
| combine model compression with representation compression rather than using either individually | CIFAR10 generalization-prediction metrics · CIFAR10 | correlation with generalization gap | Helped |
| use representation compression alone without model-compression information (ablation) | feature-compression metrics in binned models · 2D clustered classification with feature binning and IB regularization | Pearson correlation with loss generalization gap: 0.0057 | No clear effect |
Closest papers by meaning
- On the Information Bottleneck Theory of Deep Learning (ICLR 2018 · similarity 0.91)
- Information Bottleneck Analysis of Deep Neural Networks via Lossy Compression (ICLR 2024 · similarity 0.88)
- Information Bottleneck: Exact Analysis of (Quantized) Neural Networks (ICLR 2022 · similarity 0.88)
- Structured IB: Improving Information Bottleneck with Structured Feature Learning (AAAI 2025 · similarity 0.85)
- Cauchy-Schwarz Divergence Information Bottleneck for Regression (ICLR 2024 · similarity 0.85)
- Understanding and Leveraging the Learning Phases of Neural Networks (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.