AI Paper Atlas

Research map / Architectures and efficiency / RNNs and normalization

Intrinsic Information Theoretic Analysis of ReLU Nets

Johan Mylius-Kroken, Elisabeth Wetzer, Ali Ramezani-Kebrya, Robert Jenssen, Kristoffer Wickstrøm

NeurIPS 2026 · Architectures and efficiency › RNNs and normalization · approach: Width and depth of ReLU networks · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that continuous piecewise-linear networks such as ReLU nets admit an intrinsic information-theoretic analysis that avoids hyperparameter-dependent mutual information estimators.

Problem

Mutual information estimators for representation analysis in high-dimensional deep networks depend on hyperparameters (bin widths, kernel bandwidths, neighbourhood radii, auxiliary networks) whose bias often dominates the signal.

Approach

Exploits the piecewise-linear structure of CPWL networks, including ReLU nets, to perform an intrinsic information-theoretic analysis that does not require hyperparameter-dependent MI estimation.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use the hyperparameter-free routing plug-in estimator in place of binning or k-means MI estimatesMI estimation on ReLU nets · 75 networks, Composite/WBC/MNIST FCPearson r with baselines: r = 0.911 (binning), 0.922 (k-means)No clear effect
compare routing estimator with KSG estimatorrouting estimator · 75 networksPearson r: r = 0.892Mixed
apply Miller-Madow bias correction to a saturated partition (ablation)plug-in routing estimator · occupancy rho > 0.9mean bias correction (bits): 5.403 bits correctionHurt
apply Grassberger, Chao-Shen or Chao-Wang-Jost bias correction (ablation)plug-in routing estimator · occupancy rho > 0.9estimate relative to H(Y): within 0.069 bits of H(Y)No clear effect
add functional-equivalence quotient at eps 0.3 to 0.5 (ablation)routing estimator, MNIST PCA d>=10 · width 15 networksestimate below H(Y): 0.266-0.425 bits below H(Y) at eps=0.5Improved
add functional-equivalence quotient at eps 0.5 (ablation)routing estimator, MNIST PCA d>=10 · width 25 networksestimate below H(Y): 0.027-0.074 bits below H(Y)Improved
add functional-equivalence quotient at eps 0.5 (ablation)routing estimator, MNIST PCA d>=10 · width 50 networksestimate below H(Y): gap <= 0.004 bitsNo clear effect
use large tolerance eps of 1 or 2 in the functional quotient (ablation)functional quotient · PCA-reduced MNISTinformativeness: at d=2, eps=2 keeps 0.732-0.828 of the raw valueHurt
increase PCA input dimension from 5 to 10-20 for wide networks (ablation)raw routing estimator · MNIST widths 25 and 50routing information: approaches H(Y) = 3.319Hurt
train with permuted labels, narrow network (ablation)routing estimator · Composite [5,5,5]information about true labels (bits): 1.949 -> 1.411 raw; 1.828 -> 1.096 quotientImproved
train with permuted labels, wide network (ablation)routing estimator · Composite [25,25,25]information about true labels (bits): 2.433 raw, 2.329 after quotient, H(Y) = 2.639No clear effect

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy