Research map / Optimization / Nonconvex and smooth optimization
On the Stability and Generalization of Meta-Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces uniform meta-stability and uses it to derive generalization bounds for proximal (regularized ERM / GD task-specific) meta-learning on convex smooth, weakly convex non-smooth, and robust losses.
Problem
Generalization guarantees for meta-learning (transfer risk to unseen tasks) are weak or unavailable for practical bi-level, proximal meta-learning algorithms, especially with non-smooth or non-convex losses.
Approach
Defines uniform meta-stability, which measures sensitivity when both a training task and a test-task training example are replaced, and shows that stable meta-learners generalize with rate roughly log(mn)/ (m+n) style terms; instantiates it for regularized ERM and GD-based proximal meta-learning (Meta-Prox style).
Key results
- Uniformly meta-stable algorithms yield generalization gap bound of order log(mn)(1/m+1/n+1/sqrt(mn)), tighter than prior Bousquet-based rates for n > m.
- Excess transfer risk for convex smooth losses is O(1/(lambda sqrt K) + 1/(lambda m) + 1/(lambda n) + lambda/T + lambda sigma^2), converging to sigma^2 as K,T,m,n grow.
- On synthetic sine regression, empirical generalization gap decreases roughly as O(1/m + 1/n + 1/sqrt(mn)) as predicted by theory.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use uniform meta-stability to analyze generalization | Prox Meta-Learning · Meta-learning transfer risk | generalization gap: O(1/m + 1/n + 1/√(mn)) vs. O(1/m + 1/n + 1/√m) | Helped |
| use RERM for task-specific learning | Prox Meta-Learning · Convex, G-Lipschitz losses | uniform meta-stability β̄: G²/(λm) + G²/(λn) | Helped |
| use RERM for task-specific learning with smooth losses | Prox Meta-Learning · Convex, H-smooth, M-bounded losses | uniform meta-stability β̄: HM/[λ(2n−1)] + HM/[λ(m+1)] | Helped |
| use GD for task-specific learning | Prox Meta-Learning · Convex, G-Lipschitz, H-smooth losses | uniform meta-stability β̄: G²/(λm) + G²/(λn) | Helped |
| use GD for task-specific learning on weakly convex, non-smooth losses | Prox Meta-Learning · ρ-weakly convex, G-Lipschitz losses | uniform meta-stability β̄: G²√(η/λ) + G²/(λm) + G²/(λn) | Helped |
| use stochastic meta-parameter updates | Prox Meta-Learning with GD · ρ-weakly convex, G-Lipschitz losses | uniform meta-stability β̄: G²√(η/λ) + G²/(λm) + G²/(λn), w.h.p. | Helped |
| increase the number of training tasks and/or test-task samples | Prox Meta-Learning · Synthetic sine wave regression | transfer risk | Helped |
Surprising findings
- Generalization bounds are independent of the number of meta-learning and inner GD iterations due to strong convexity of the regularized objective.
Limitations
- Bounds are not shown to be tight; no lower bounds on meta-generalization error.
- Experiments are only a small synthetic sine-wave check, not real benchmarks.
- Weakly convex results require regularization lambda >= 2 rho, which must be tuned.
Experimental setup
Datasets: synthetic sine wave regression tasks
Benchmarks: synthetic sine wave regression
Metrics: generalization gap transfer risk mean squared error
Baselines: Maurer 2005 bound (Theorem 2.1) Bousquet et al. 2020 bound (Theorem 2.2) Denevi et al. 2019a bound Guan et al. 2022
Builds on: Maurer 2005 meta-learning stability Bousquet et al. 2020 sharper stability bounds Meta-Prox (Zhou et al. 2019) Guan et al. 2022 episodic stability Fallah et al. 2021
Closest papers by meaning
- Fine-Grained Analysis of Stability and Generalization for Modern Meta Learning Algorithms (NeurIPS 2022 · similarity 0.94)
- Generalization Bounds for Meta-Learning via PAC-Bayes and Uniform Stability (NeurIPS 2021 · similarity 0.91)
- Provable Guarantees for Gradient-Based Meta-Learning (ICML 2019 · similarity 0.90)
- On the Stability and Generalization of Meta-Learning: the Impact of Inner-Levels (NeurIPS 2025 · similarity 0.90)
- Theoretical Investigations and Practical Enhancements on Tail Task Risk Minimization in Meta Learning (NeurIPS 2024 · similarity 0.88)
- Stability and Generalization of Stochastic Compositional Gradient Descent Algorithms (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.