Research map / Probabilistic and scientific modeling / Variational and generative sampling
Generalization Bounds via Meta-Learned Model Representations: PAC-Bayes and Sample Compression Hypernetworks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes three hypernetwork meta-learners (PAC-Bayes encoder, sample-compression, and a hybrid) that output downstream predictors with per-predictor generalization certificates, backed by a new PAC-Bayes sample compression theorem for continuous messages.
Problem
Obtaining non-vacuous, per-predictor generalization bounds for predictors produced by meta-learned hypernetworks from a dataset input.
Approach
Use an explicit information bottleneck in the hypernetwork (a PAC-Bayes Gaussian latent, a sample compressor selecting c examples plus an optional message, or a hybrid with a continuous message), and bound each generated predictor using the bottleneck's complexity via a new sample compression theorem allowing real-valued losses and continuous messages.
Key results
- On binary MNIST, SCH+ achieves a bound of 0.280 with test error 0.155, versus 0.767 bound for the best PAC-Bayes benchmark baseline in that table row (bound ~0.767 with test error 0.369).
- On 100-pixel swap, SCH+ obtains the smallest bound of 0.035 but with test error 0.024, while the PAC-Bayes Catoni baseline has test error 0.015.
- Benchmark PAC-Bayes methods yield KL values of 14 to 661 on binary MNIST, indicating uninformative bounds where the proposed approaches stay informative.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use a learned binary message in addition to the compression set (SCH+ vs SCH-) (ablation) | Sample Compression Hypernetwork · MNIST-pixels-swap, 100 Pixels swap | Bound: 0.067 ± 0.007 -> 0.035 ± 0.012 | Helped |
| use a learned binary message in addition to the compression set (SCH+ vs SCH-) (ablation) | Sample Compression Hypernetwork · MNIST-pixels-swap, 200 Pixels swap | Bound: 0.129 ± 0.023 -> 0.091 ± 0.022 | Helped |
| use a learned binary message in addition to the compression set (SCH+ vs SCH-) (ablation) | Sample Compression Hypernetwork · MNIST-pixels-swap, 300 Pixels swap | Bound: 0.193 ± 0.038 -> 0.177 ± 0.032 | Helped |
| use a learned binary message in addition to the compression set (SCH+ vs SCH-) (ablation) | Sample Compression Hypernetwork · MNIST-pixels-swap | Test error: 0.029 ± 0.007 -> 0.024 ± 0.005; 0.084 ± 0.017 -> 0.075 ± 0.019; 0.162 ± 0.043 -> 0.153 ± 0.028 | Mixed |
| add a binary message to the sample-compression hypernetwork (ablation) | Sample Compression Hypernetwork · Binary MNIST | Bound: 0.352 ± 0.187 -> 0.280 ± 0.148 | Helped |
| add a binary message to the sample-compression hypernetwork (ablation) | Sample Compression Hypernetwork · Binary MNIST | Test error: 0.278 ± 0.076 -> 0.155 ± 0.109 | Helped |
| add a binary message to the sample-compression hypernetwork (ablation) | Sample Compression Hypernetwork · Binary CIFAR100 | Bound: 0.600 ± 0.143 -> 0.745 ± 0.101 | Hurt |
| add a binary message to the sample-compression hypernetwork (ablation) | Sample Compression Hypernetwork · Binary CIFAR100 | Test error: 0.374 ± 0.118 -> 0.305 ± 0.142 | Hurt |
| use an opaque encoder that ignores the task input rather than a task-conditioned representation (ablation) | Meta-learning hypernetwork · Binary MNIST and CIFAR100 | Test error: 0.497 ± 0.134 -> 0.150 ± 0.114; 0.506 ± 0.101 -> 0.295 ± 0.103 | Hurt |
| use task-conditioned representations rather than an opaque encoder (ablation) | Meta-learning hypernetwork · MNIST-pixels-swap | Test error: opaque encoder 0.037 ± 0.006; 0.087 ± 0.018; 0.159 ± 0.031 | No clear effect |
Surprising findings
- On the MNIST pixel-swap tasks the proposed methods cannot learn task-specific information and perform similar to an opaque constant-output strawman, while benchmark PAC-Bayes methods do well.
- The authors identify a small error in a previously published sample compression theorem (Lever et al.), noting it only holds for compression set size equal to lambda.
Limitations
- On MNIST pixel-swap tasks, the proposed hypernetworks perform similarly to a constant opaque encoder and do not exploit task-environment differences.
- Bounds for the Dirac sample compression and continuous-message variants depend on the compression set and message sizes, with a trade-off that requires tuning.
- Experiments are limited to small fully connected downstream networks and simple datasets.
Experimental setup
Datasets: moons (scikit-learn) MNIST CIFAR100
Benchmarks: MNIST pixel-swap (100/200/300) binary MNIST binary CIFAR100
Metrics: generalization bound test error KL value
Baselines: PAC-Bayesian meta-learning bounds (Rothfuss et al.) Pentina and Lampert style PAC-Bayes meta-learning Opaque encoder
Builds on: PAC-Bayes sample compression (Lever et al.) General PAC-Bayes theorem (Begin et al.) Set Covering Machine sample compression bound (Marchand and Shawe-Taylor) DeepSets Straight-through estimator
Closest papers by meaning
- PAC-Bayes Compression Bounds So Tight That They Can Explain Generalization (NeurIPS 2022 · similarity 0.92)
- Non-vacuous Generalization Bounds at the ImageNet Scale: a PAC-Bayesian Compression Approach (ICLR 2019 · similarity 0.91)
- Stronger Generalization Bounds for Deep Nets via a Compression Approach (ICML 2018 · similarity 0.90)
- Deterministic PAC-Bayesian generalization bounds for deep networks via generalizing noise-resilience (ICLR 2019 · similarity 0.90)
- Does Compression Imply Generalization? A Minimum Description Length Perspective (NeurIPS 2026 · similarity 0.88)
- Slicing Mutual Information Generalization Bounds for Neural Networks (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.