Research map / Learning theory and algorithms / Kernels and regression theory
Slicing Mutual Information Generalization Bounds for Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives tighter input-output mutual information generalization bounds for neural networks trained on random subspaces (slicing), using disintegrated and k-sliced MI, extended to approximately compressible weights via rate-distortion, and evaluated on MNIST/CIFAR-10 MLPs.
Problem
Input-output mutual information generalization bounds are vacuous or intractable to estimate for high-dimensional neural network weights.
Approach
Train on random d-dimensional subspaces (random slicing) and bound generalization via disintegrated MI conditioned on the projection Theta, which relates to k-sliced MI and is estimable in low dimension; extend via rate-distortion to weights only approximately on the subspace, with a distortion regularizer.
Key results
- Slicing followed by quantization yields non-vacuous information-theoretic generalization bounds for fully connected NNs on MNIST and CIFAR-10, a task previously out of reach.
- Gaussian mean estimation: the disintegrated bound applies for d<D where prior bounds do not, and bounds decrease as d decreases.
- Regularizing the distortion term with lambda (e.g., (lambda, d)=(10, 1000)) gives tighter rate-distortion bounds while maintaining satisfactory train and test accuracy on MNIST.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train on a lower-dimensional random subspace (ablation) | Random-subspace learning algorithms · Gaussian mean estimation | generalization error | Helped |
| Train on a lower-dimensional random subspace (ablation) | Logistic regression · Binary classification with Gaussian data | test risk | Hurt |
| Train on a lower-dimensional random subspace (ablation) | Fully connected neural networks · MNIST and CIFAR-10 | test performance | Mixed |
| Add regularization encouraging weights to lie near the random subspace (ablation) | Feedforward neural network · MNIST | generalization error | Helped |
| Increase regularization encouraging weights to lie near the random subspace (ablation) | Feedforward neural network · MNIST | train and test risk | Hurt |
| Use a smaller subspace dimension and increase compressibility regularization (ablation) | Feedforward neural network · MNIST | generalization bound and train/test performance | Mixed |
| Use disintegration and individual-sample disintegrated mutual information instead of the standard input-output MI bound | Random-subspace learning algorithms · Generalization bounds | bound tightness | Helped |
| Combine random slicing with quantization | Fully connected neural networks · MNIST and CIFAR-10 | computability of generalization bounds | Helped |
| Use quantization without slicing (ablation) | Neural networks · High-dimensional image classification | computability of generalization bounds | Hurt |
| Use the individual-sample disintegrated-MI bound rather than the full-training-set disintegrated-MI bound | Random-subspace learning algorithms · Generalization bounds | bound applicability | Helped |
| Apply the random-subspace bound at d < D rather than using the prior bound that applies only at d = D | Gaussian mean estimation · Generalization bounds | bound applicability | Helped |
Surprising findings
- kNN-based MI estimation produced NaN values for d>2, so only MINE-based bounds are reported.
- Quantization alone is far from sufficient for non-vacuous bounds in high dimensions; slicing is the key ingredient.
Limitations
- Bounds remain looser than those from data-dependent PAC-Bayes or conditional MI frameworks.
- Expectation over Theta approximated by Monte Carlo; MI terms for MNIST are estimated with MINE, and rate-distortion bounds use an entropy-based quantization surrogate.
- Experiments are limited to small fully-connected networks on MNIST, CIFAR-10 and Iris.
Experimental setup
Datasets: MNIST CIFAR-10 Iris synthetic Gaussian data
Benchmarks: MNIST CIFAR-10
Metrics: generalization error generalization bound value train/test risk train/test accuracy distortion term MI term
Baselines: standard MI generalization bound (Xu & Raginsky) individual-sample MI bound (Bu et al.) PAC-Bayes intrinsic dimension bound (Lotfi et al.)
Builds on: information-theoretic generalization bounds (Xu & Raginsky) individual-sample MI bounds (Bu et al.) intrinsic dimension of objective landscapes (Li et al.) sliced mutual information MINE
Closest papers by meaning
- Tighter CMI-Based Generalization Bounds via Stochastic Projection and Quantization (NeurIPS 2025 · similarity 0.89)
- Fantastic Generalization Measures are Nowhere to be Found (ICLR 2024 · similarity 0.89)
- Towards Generalization beyond Pointwise Learning: A Unified Information-theoretic Perspective (ICML 2024 · similarity 0.88)
- Generalizability of Neural Networks Minimizing Empirical Risk Based on Expressive Power (ICLR 2025 · similarity 0.88)
- Exactly Tight Information-theoretic Generalization Bounds via Binary Jensen-Shannon Divergence (ICML 2025 · similarity 0.88)
- Generalization Bounds via Meta-Learned Model Representations: PAC-Bayes and Sample Compression Hypernetworks (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.