Research map / Learning theory and algorithms / Kernels and regression theory
Entropy and mutual information in models of deep neural networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use bounded hardtanh instead of an unbounded activation while increasing weight magnitude | Two-layer stochastic network · Entropy versus weight scaling | entropy | Mixed |
| Use ReLU-ReLU activations rather than a linear network | Two-layer stochastic network · Entropy versus weight scaling | entropy: half of the linear network entropy | Hurt |
| Estimate entropy with non-parametric methods instead of the replica computation | Two-layer stochastic network · Linear and hardtanh networks | entropy estimation | Hurt |
| Use the replica computation that incorporates the network structure | Two-layer stochastic network · ReLU-ReLU network | entropy estimation | Helped |
| Constrain layers to USV form, keeping the orthogonal factors fixed and updating only the diagonal matrix | Linear networks trained with USV-layers · Mutual-information estimation during training | squared estimation error | Helped |
| Train the linear student with plain SGD | 4-layer linear student · Linear teacher-student regression | mutual information between input and hidden layers | No clear effect |
| Train the recognition model with ReLU activations | Deep recognition model · Correlated-input binary classification | mutual information between input and hidden layers | Mixed |
| Train the recognition model with hardtanh activations | Deep recognition model · Correlated-input binary classification | mutual information between input and hidden layers | Mixed |
| Relate compression during training to deeper layers | Deep recognition models · Correlated-input binary classification | compression by depth | No clear effect |
| Relate compression to generalization | Deep recognition models · Correlated-input binary classification | generalization | No clear effect |
| Add interleaved linear layers to the ReLU recognition network | Deep recognition model with ReLU · Correlated-input binary classification | training success | Helped |
Closest papers by meaning
- Information Bottleneck Analysis of Deep Neural Networks via Lossy Compression (ICLR 2024 · similarity 0.87)
- Neural Thermodynamics: Entropic Forces in Deep and Universal Representation Learning (NeurIPS 2025 · similarity 0.87)
- Neural Entropy (NeurIPS 2025 · similarity 0.87)
- MINDE: Mutual Information Neural Diffusion Estimation (ICLR 2024 · similarity 0.86)
- InfoNet: Neural Estimation of Mutual Information without Test-Time Optimization (ICML 2024 · similarity 0.85)
- Beyond IID weights: sparse and low-rank deep Neural Networks are also Gaussian Processes (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.