Research map / Architectures and efficiency / RNNs and normalization
Intrinsic Information Theoretic Analysis of ReLU Nets
Johan Mylius-Kroken, Elisabeth Wetzer, Ali Ramezani-Kebrya, Robert Jenssen, Kristoffer Wickstrøm
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that continuous piecewise-linear networks such as ReLU nets admit an intrinsic information-theoretic analysis that avoids hyperparameter-dependent mutual information estimators.
Problem
Mutual information estimators for representation analysis in high-dimensional deep networks depend on hyperparameters (bin widths, kernel bandwidths, neighbourhood radii, auxiliary networks) whose bias often dominates the signal.
Approach
Exploits the piecewise-linear structure of CPWL networks, including ReLU nets, to perform an intrinsic information-theoretic analysis that does not require hyperparameter-dependent MI estimation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use the hyperparameter-free routing plug-in estimator in place of binning or k-means MI estimates | MI estimation on ReLU nets · 75 networks, Composite/WBC/MNIST FC | Pearson r with baselines: r = 0.911 (binning), 0.922 (k-means) | No clear effect |
| compare routing estimator with KSG estimator | routing estimator · 75 networks | Pearson r: r = 0.892 | Mixed |
| apply Miller-Madow bias correction to a saturated partition (ablation) | plug-in routing estimator · occupancy rho > 0.9 | mean bias correction (bits): 5.403 bits correction | Hurt |
| apply Grassberger, Chao-Shen or Chao-Wang-Jost bias correction (ablation) | plug-in routing estimator · occupancy rho > 0.9 | estimate relative to H(Y): within 0.069 bits of H(Y) | No clear effect |
| add functional-equivalence quotient at eps 0.3 to 0.5 (ablation) | routing estimator, MNIST PCA d>=10 · width 15 networks | estimate below H(Y): 0.266-0.425 bits below H(Y) at eps=0.5 | Improved |
| add functional-equivalence quotient at eps 0.5 (ablation) | routing estimator, MNIST PCA d>=10 · width 25 networks | estimate below H(Y): 0.027-0.074 bits below H(Y) | Improved |
| add functional-equivalence quotient at eps 0.5 (ablation) | routing estimator, MNIST PCA d>=10 · width 50 networks | estimate below H(Y): gap <= 0.004 bits | No clear effect |
| use large tolerance eps of 1 or 2 in the functional quotient (ablation) | functional quotient · PCA-reduced MNIST | informativeness: at d=2, eps=2 keeps 0.732-0.828 of the raw value | Hurt |
| increase PCA input dimension from 5 to 10-20 for wide networks (ablation) | raw routing estimator · MNIST widths 25 and 50 | routing information: approaches H(Y) = 3.319 | Hurt |
| train with permuted labels, narrow network (ablation) | routing estimator · Composite [5,5,5] | information about true labels (bits): 1.949 -> 1.411 raw; 1.828 -> 1.096 quotient | Improved |
| train with permuted labels, wide network (ablation) | routing estimator · Composite [25,25,25] | information about true labels (bits): 2.433 raw, 2.329 after quotient, H(Y) = 2.639 | No clear effect |
Closest papers by meaning
- Understanding Deep Neural Networks with Rectified Linear Units (ICLR 2018 · similarity 0.87)
- On the Local Complexity of Linear Regions in Deep ReLU Networks (ICML 2025 · similarity 0.87)
- Most ReLU Networks Admit Identifiable Parameters (NeurIPS 2026 · similarity 0.86)
- Information Bottleneck Analysis of Deep Neural Networks via Lossy Compression (ICLR 2024 · similarity 0.86)
- Expressivity of ReLU-Networks under Convex Relaxations (ICLR 2024 · similarity 0.86)
- Make Haste Slowly: A Theory of Emergent Structured Mixed Selectivity in Feature Learning ReLU Networks (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
