Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
A Hierarchical Bayesian Model for Few-Shot Meta Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NIW-Meta, a hierarchical Bayesian few-shot meta-learning method with a Normal-Inverse-Wishart prior/posterior that enables one-time episodic optimization via SGLD-based quadratic approximation and closed-form local posteriors.
Problem
Few-shot meta learning with a large or infinite number of episodes requires a principled Bayesian treatment of episode-specific network weights, but conventional variational inference cannot store per-episode posteriors and MAML-style methods are memory-heavy.
Approach
Model episode-specific network weights as local random variables governed by a global NIW-distributed variable; approximate each episodic loss quadratically using SGLD sample statistics so that local posteriors have closed forms, letting the global variational parameters be learned online with one-time episodic updates and no unrolled computation graph.
Key results
- On miniImageNet with WRN-28-10, NIW-Meta reaches 68.54% 1-shot and 84.81% 5-shot, above MetaQDA's 67.83% and 84.28%.
- On Sine-Line regression, NIW-Meta achieves MSE 0.7822 with R-ECE 0.1728, versus 0.9160 and 0.2666 for PMAML.
- On miniImageNet with WRN-28-10, ECE drops to 10.79 (1-shot) versus 30.68 for MetaQDA-FB, before temperature scaling.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use NIW-Meta instead of MetaQDA | Few-shot classification with ResNet-18 · miniImageNet 1-shot | accuracy: 65.12 ± 0.66 -> 65.49 ± 0.56 | Helped |
| Use NIW-Meta instead of MetaQDA | Few-shot classification with WRN-28-10 · miniImageNet 5-shot | accuracy: 84.28 ± 0.69 -> 84.81 ± 0.28 | Helped |
| Use NIW-Meta instead of MetaQDA | Few-shot classification with ResNet-18 · tieredImageNet 1-shot | accuracy: 69.97 ± 0.52 -> 70.52 ± 0.19 | Helped |
| Use NIW-Meta instead of ProtoNet | Few-shot classification with DINO/s · miniImageNet 1-shot | accuracy: 93.1 -> 93.4 | Helped |
| Use NIW-Meta instead of FEAT | FEAT architecture for few-shot classification · miniImageNet 5-shot | accuracy: 82.05 -> 82.28^{± 0.15} | Helped |
| Use NIW-Meta instead of MetaQDA-FB | Few-shot classification with Conv-4 · miniImageNet 1-shot | ECE: 4.32 -> 2.68 | Helped |
| Use NIW-Meta instead of MAML (1-step) | Few-shot regression with Ridge head · Sine-Line 5-shot | mean squared error: 0.8206 -> 0.7822 | Helped |
| Use NIW-Meta instead of PMAML (1-step) | Few-shot regression with Ridge head · Sine-Line 5-shot | R-ECE: 0.2666 -> 0.1728 | Helped |
| Add NIW-Meta to the Conv-net feature extractor and Ridge head | Pose estimation · Pascal-1D | test error: 1.02 ± 0.06 -> 0.89 ± 0.06 | Helped |
| Add NIW-Meta to the ANP architecture | Pose estimation · ShapeNet-1D intra-category | test error: 5.81 ± 0.23 -> 5.47 ± 0.12 | Helped |
| Use a Conv-net feature extractor with Ridge head instead of CNP/ANP | NIW-Meta · ShapeNet-2D and Distractor | test error: ShapeNet-2D IC: 13.74^{± 0.30} (ANP) -> 21.25^{± 0.76} (C+R); Distractor IC: 1.80^{± 0.01} (CNP) -> 8.90^{± 0.26} (C+R) | Hurt |
Surprising findings
- On ShapeNet-2D and Distractor, the conv-net plus ridge head variant performed poorly, which the authors attribute to heavy noise and distractors requiring interaction modeling.
Limitations
- Relies on a quadratic approximation of the episodic loss via a few SGLD iterations, which is an approximation.
- Test-time prediction uses a mode approximation and variational inference on the support set, adding overhead.
- Theoretical generalization bounds rely on strong assumptions (MLP backbone, bounded parameters, regression setting).
Experimental setup
Datasets: miniImageNet tieredImageNet CIFAR-FS Sine-Line Pascal-1D ShapeNet-1D ShapeNet-2D Distractor
Benchmarks: miniImageNet tieredImageNet CIFAR-FS Sine-Line Pascal-1D ShapeNet-1D ShapeNet-2D Distractor
Metrics: accuracy expected calibration error mean squared error regression ECE pose estimation error
Baselines: MAML ProtoNet MetaQDA SimpleShot S2M2 MetaOpt FEAT PMAML CNP ANP CNP+BA CNP+FCL
Builds on: MAML ProtoNet Reptile PLATIPUS MetaQDA
Closest papers by meaning
- MELR: Meta-Learning via Modeling Episode-Level Relationships for Few-Shot Learning (ICLR 2021 · similarity 0.89)
- Recasting Gradient-Based Meta-Learning as Hierarchical Bayes (ICLR 2018 · similarity 0.87)
- Bayesian Meta-Learning for the Few-Shot Setting via Deep Kernels (NeurIPS 2020 · similarity 0.87)
- Amortized Bayesian Meta-Learning (ICLR 2019 · similarity 0.87)
- MARS: Meta-learning as Score Matching in the Function Space (ICLR 2023 · similarity 0.87)
- More Flexible PAC-Bayesian Meta-Learning by Learning Learning Algorithms (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.