Research map / Learning theory and algorithms / Kernels and regression theory
More Flexible PAC-Bayesian Meta-Learning by Learning Learning Algorithms
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a PAC-Bayesian meta-learning framework whose bounds are over distributions of learning algorithms (not just priors), yielding bounds that cover hypernetwork, representation, optimizer and hybrid initialization+regularization meta-learners.
Problem
Existing PAC-Bayesian meta-learning bounds only apply to methods that transfer knowledge via prior distributions over models, excluding representation-, hypernetwork-, and optimization-based meta-learners.
Approach
Model knowledge transfer as a meta-posterior over learning algorithms (rather than priors), with algorithm-specific hyper-posteriors over priors, and prove generalization bounds for this general setup; the prior-based setting is recovered as a special case and the bounds are tighter than prior ones.
Key results
- On permuted labels, the framework reaches 7.91% test error versus 13.7% for the prior-based Pentina & Lampert-style baseline, while on shuffled pixels it matches it (9.9% vs 9.92%).
- In the linear-classifier setting (d=2, m=5), the new bound is the tightest of the compared bounds and becomes non-vacuous already at n=10 tasks, while others remain vacuous for n<20.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Learn the initialization separately from the regularization prior | Prior-based meta-learning on MNIST permuted labels · MNIST permuted labels (PL), test | test error (%): 13.7 -> 7.91 | Helped |
| Learn the initialization separately from the regularization prior | Prior-based meta-learning on MNIST shuffled pixels · MNIST shuffled pixels (SP), test | test error (%): 9.92 -> 9.9 | No clear effect |
Surprising findings
- Separating initialization from regularization prior gives no benefit on shuffled pixels, where the learned parameters were almost identical.
Limitations
- Empirical evaluation is limited to two MNIST-based settings and a small linear-classifier experiment.
- Bounds for the hybrid initialization+regularization case are only demonstrated with a simple two-distribution algorithm family.
Experimental setup
Datasets: MNIST
Benchmarks: MNIST shuffled pixels (SP) MNIST permuted labels (PL) synthetic linear classifier task environment
Metrics: test error (%) PAC-Bayes bound value (empirical loss plus complexity)
Baselines: Pentina & Lampert prior-based PAC-Bayes meta-learning Amit & Meir meta-learning bound independent learning
Builds on: PAC-Bayesian meta-learning (Pentina & Lampert) PAC-Bayes bounds for meta-learning (Amit & Meir) Baxter's learning-to-learn framework change of measure inequality
Closest papers by meaning
- Meta-Learning by Adjusting Priors Based on Extended PAC-Bayes Theory (ICML 2018 · similarity 0.94)
- PACOH: Bayes-Optimal Meta-Learning with PAC-Guarantees (ICML 2021 · similarity 0.93)
- Bridging the Gap Between Practice and PAC-Bayes Theory in Few-Shot Meta-Learning (NeurIPS 2021 · similarity 0.92)
- A Unified View on PAC-Bayes Bounds for Meta-Learning (ICML 2022 · similarity 0.92)
- Recursive PAC-Bayes: A Frequentist Approach to Sequential Prior Updates with No Information Loss (NeurIPS 2024 · similarity 0.89)
- Learning via Surrogate PAC-Bayes (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.