Research map / Architectures and efficiency / RNNs and normalization
Information Bottleneck: Exact Analysis of (Quantized) Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Compute MI exactly in 8-bit quantization-aware trained networks instead of estimating it by post-hoc binning | Tanh and ReLU synthetic-data networks · 12-bit synthetic data | MI measurement validity | Helped |
| Use Tanh rather than ReLU hidden activations (ablation) | 8-bit quantized synthetic-data network · 12-bit synthetic data | hidden-layer compression | Mixed |
| Use ReLU rather than Tanh hidden activations (ablation) | 8-bit quantized synthetic-data network · 12-bit synthetic data | hidden-layer compression | Hurt |
| Use a two-neuron bottleneck | 8-bit quantized ReLU network · MNIST | compression phase | Helped |
| Replace the two-neuron bottleneck with a wider bottleneck or no bottleneck (ablation) | 8-bit quantized ReLU network · MNIST | non-trivial information curves | Hurt |
| Use a four-neuron rather than two-neuron bottleneck (ablation) | 8-bit quantized ReLU network · MNIST | MI measurements | Mixed |
| Expand the network after a two-neuron bottleneck (ablation) | 8-bit quantized ReLU network · MNIST | information retained after bottleneck | No clear effect |
| Use a 4x10 or convolutional architecture without a bottleneck (ablation) | 8-bit quantized ReLU network · MNIST | non-trivial information curves | Hurt |
| Reduce activation precision from 8 bits to 4 bits (ablation) | Tanh and ReLU networks · 12-bit synthetic data | network accuracy | Hurt |
| Reduce activation precision from 8 bits to 4 bits (ablation) | Tanh network · 12-bit synthetic data | distinguishable layers in the information plane | Helped |
| Increase precision from 8 bits to 32 bits (ablation) | Tanh and ReLU networks · 12-bit synthetic data | information loss between layers | Hurt |
| Estimate MI with 30, 100, or 256 bins rather than compute exact MI (ablation) | Tanh and ReLU networks · 12-bit synthetic data | MI estimate stability | Hurt |
Closest papers by meaning
- Information Bottleneck Analysis of Deep Neural Networks via Lossy Compression (ICLR 2024 · similarity 0.95)
- On the Information Bottleneck Theory of Deep Learning (ICLR 2018 · similarity 0.92)
- Understanding and Leveraging the Learning Phases of Neural Networks (AAAI 2024 · similarity 0.89)
- Cauchy-Schwarz Divergence Information Bottleneck for Regression (ICLR 2024 · similarity 0.88)
- Structured IB: Improving Information Bottleneck with Structured Feature Learning (AAAI 2025 · similarity 0.87)
- A Rate-Distortion View of Uncertainty Quantification (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.