Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
Accelerating Convergence in Bayesian Few-Shot Classification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MD-BFSC, which applies mirror descent-based variational inference to deep-kernel Gaussian process few-shot classification, turning non-conjugate inference into conjugate Bayesian updates without auxiliary variables.
Problem
Non-conjugate posterior inference in Gaussian process multi-class classification for few-shot learning, where existing GP-based methods either use a misspecified Gaussian likelihood or need auxiliary variables, and first-order bi-level optimization converges slowly.
Approach
Replace natural gradient descent in the inner-loop variational inference with mirror descent over the mean parameter using the negative-entropy Bregman divergence; this is shown equivalent to natural gradient on the natural parameter, needs only first-order gradients, and reduces each step to a conjugate Gaussian update with a Gaussian approximation of the softmax likelihood.
Key results
- MD-BFSC achieves 65.45% 1-shot accuracy on CUB (in-domain) and 40.75% 1-shot on mini-ImageNet to CUB, the best in those settings.
- MD-BFSC attains the lowest 5-shot ECE on CUB (0.005) and mini-ImageNet to CUB (0.005), and lowest MCE on mini-ImageNet to CUB (0.014) and Omniglot to EMNIST (0.025 reported in table, 0.024 in text).
- In inner-loop experiments, mirror descent's ELBO increases faster than vanilla gradient descent over 30 steps on CUB and mini-ImageNet in 1-shot and 5-shot settings.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use mirror descent instead of vanilla gradient descent in the inner loop | GP-based few-shot classification · CUB and mini-ImageNet, 1-shot and 5-shot | ELBO convergence | Helped |
| use mirror descent instead of vanilla gradient descent in the inner loop | GP-based few-shot classification · CUB and mini-ImageNet, outer-loop 1-shot and 5-shot | cross-entropy loss convergence | Helped |
| use MD-BFSC instead of the strongest reported baseline | few-shot classification · CUB 1-shot and mini-ImageNet → CUB 1-shot | accuracy: CDKT (ML) (τ < 1): 65.21 ± 0.45 vs 65.45 ± 0.42; 40.43 ± 0.43 vs 40.75 ± 0.31 | Helped |
| use MD-BFSC instead of the strongest reported baseline | few-shot classification · CUB 5-shot, mini-ImageNet → CUB 5-shot, and Omniglot → EMNIST 1-shot and 5-shot | accuracy: 78.38 ± 0.21 vs 79.10 ± 0.33; 56.98 ± 0.30 vs 57.31 ± 0.11; 74.02 ± 0.49 vs 77.00 ± 0.50; 87.05 ± 0.23 vs 88.10 ± 0.78 | Hurt |
| use MD-BFSC instead of the strongest reported baseline | few-shot classification · 5-shot CUB, mini-ImageNet → CUB, and Omniglot → EMNIST | ECE: 0.005; 0.005; 0.010 | Mixed |
| use MD-BFSC instead of the strongest reported baseline | few-shot classification · 5-shot CUB, mini-ImageNet → CUB, and Omniglot → EMNIST | MCE: 0.067; 0.014; 0.025 | Mixed |
| increase inner-loop step size from ρ=0.01 to ρ=1 (ablation) | MD-BFSC · CUB 1-shot 5-way | accuracy: 32.22 ± 0.41 -> 65.45 ± 0.42 | Helped |
| replace COS with RBF kernel (ablation) | MD-BFSC · CUB 1-shot 5-way | accuracy: 65.45 ± 0.42 -> 62.75 ± 0.39 | Hurt |
| replace COS with POL1 or POL2 kernel (ablation) | MD-BFSC · CUB 1-shot 5-way | accuracy: 65.45 ± 0.42 -> 64.69 ± 0.36; 65.45 ± 0.42 -> 64.74 ± 0.25 | Hurt |
| increase inner-loop steps from 3 to 1, 5, or 7 (ablation) | MD-BFSC · CUB 1-shot 5-way | accuracy: 65.45 ± 0.42 -> 62.63 ± 0.26; 65.45 ± 0.42 -> 64.18 ± 0.52; 65.45 ± 0.42 -> 62.55 ± 0.46 | Hurt |
Surprising findings
- Per-epoch wall-clock time is nearly identical to first-order baselines (about 12.1 s vs 12.1 s for gradient descent), so the claimed second-order benefit does not translate into large runtime gains.
- Performance on the MD-BFSC row for CUB 5-shot and several other settings is only comparable to, not better than, prior methods such as CDKT and OVE.
Limitations
- Outer-loop (neural network and kernel hyperparameter) updates remain first-order due to computational and memory constraints.
- Accuracy is only comparable to state-of-the-art rather than consistently superior; the best results are in a few settings.
- The softmax likelihood is approximated by a Gaussian in the conjugate update, and the Monte Carlo integral for prediction is intractable and requires sampling.
Experimental setup
Datasets: Caltech-UCSD Birds mini-ImageNet Omniglot EMNIST
Benchmarks: CUB mini-ImageNet to CUB Omniglot to EMNIST
Metrics: accuracy ECE MCE ELBO convergence cross-entropy loss convergence
Baselines: Feature Transfer Baseline++ MatchingNet ProtoNet RelationNet MAML DKT Bayesian MAML ABML OVE PG GP CDKT
Builds on: DKT MAML natural gradient descent one-vs-each softmax approximation Polya-Gamma augmentation
Closest papers by meaning
- Bayesian Few-Shot Classification with One-vs-Each Pólya-Gamma Augmented Gaussian Processes (ICLR 2021 · similarity 0.89)
- Bayesian Exploration of Pre-trained Models for Low-shot Image Classification (CVPR 2024 · similarity 0.87)
- Meta-Learning With Differentiable Convex Optimization (CVPR 2019 · similarity 0.87)
- MOKD: Cross-domain Finetuning for Few-shot Classification via Maximizing Optimized Kernel Dependence (ICML 2024 · similarity 0.85)
- Discriminative Sample-Guided and Parameter-Efficient Feature Space Adaptation for Cross-Domain Few-Shot Learning (CVPR 2024 · similarity 0.85)
- Scaling Few-Shot Learning for the Open World (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.