Research map / Learning theory and algorithms / Kernels and regression theory
Generalization Guarantees for Representation Learning via Data-Dependent Gaussian Mixture Priors
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives in-expectation and tail generalization bounds for encoder-decoder representation learning in terms of a data-dependent symmetric prior's MDL, and uses a learned Gaussian mixture prior (with attention-like weights) as a regularizer that beats VIB and CDVIB.
Problem
Generalization guarantees for representation learning (encoder-decoder) classifiers; existing bounds are vacuous or scale poorly, and MI-based bounds fail for deterministic encoders.
Approach
Bounds the generalization gap via the relative entropy between latent representations of training and ghost datasets and a symmetric, label-dependent data-dependent prior (MDL), then learns a per-label Gaussian mixture prior along training and uses its KL as a regularizer; the EM-like responsibility update resembles weighted attention.
Key results
- On ResNet18 CIFAR100, GM-MDL reaches 0.497 test accuracy vs 0.463 for CDVIB and 0.458 for VIB.
- On CNN4 CIFAR10, GM-MDL reaches 0.681 vs 0.649 for CDVIB and 0.626 for VIB.
- Theoretical in-expectation bound scales roughly as MDL(Q)/n in realizable settings, versus sqrt(MDL(Q)/n) for prior bound of the cited work.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GM-MDL instead of CDVIB | CNN4 · CIFAR10 | test accuracy: 0.649 -> 0.681 | Helped |
| Use GM-MDL instead of CDVIB | CNN4 · USPS | test accuracy: 0.955 -> 0.963 | Helped |
| Use GM-MDL instead of CDVIB | CNN4 · INTEL | test accuracy: 0.763 -> 0.776 | Helped |
| Use GM-MDL instead of CDVIB | ResNet18 · CIFAR10 | test accuracy: 0.835 -> 0.848 | Helped |
| Use GM-MDL instead of CDVIB | ResNet18 · CIFAR100 | test accuracy: 0.463 -> 0.497 | Helped |
| Use the lossy rather than the lossless Gaussian mixture prior regularizer (ablation) | Gaussian mixture MDL · Experiments across datasets | population accuracy | Helped |
Limitations
- Symmetry requirement on the prior is strict; relaxations incur penalties.
- Bounds use a distorted/quantized encoder for lossy case, and the KL between Gaussian and Gaussian mixture is only estimated via variational bounds.
- Experiments are limited to image classification with small encoders.
Experimental setup
Datasets: CIFAR10 CIFAR100 INTEL USPS
Benchmarks: CIFAR10 CIFAR100 INTEL USPS
Metrics: test accuracy
Baselines: no regularizer VIB CDVIB
Builds on: Variational Information Bottleneck Category-dependent VIB Donsker-Varadhan change of measure k-means++
Closest papers by meaning
- Connecting Jensen–Shannon and Kullback–Leibler Divergences: A New Bound for Representation Learning (NeurIPS 2025 · similarity 0.86)
- Generalization Bounds via Meta-Learned Model Representations: PAC-Bayes and Sample Compression Hypernetworks (ICML 2025 · similarity 0.86)
- Guarantees for Nonlinear Representation Learning: Non-identical Covariates, Dependent Data, Fewer Samples (ICML 2024 · similarity 0.86)
- Towards Generalization beyond Pointwise Learning: A Unified Information-theoretic Perspective (ICML 2024 · similarity 0.86)
- Generalization Analysis for Supervised Contrastive Representation Learning under Non-IID Settings (ICML 2025 · similarity 0.86)
- Information-theoretic Generalization Analysis for VQ-VAEs: A Role of Latent Variables (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.