Research map / Label-efficient and robust learning / Zero-/few-shot and meta-learning

Enabling Few-Shot Learning with PID Control: A Layer Adaptive Optimizer

ICML 2024 · Label-efficient and robust learning › Zero-/few-shot and meta-learning · approach: Meta-learning analysis and benchmarks

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes LA-PID, a MAML-based inner-loop optimizer that uses a PID-style update with layer-wise, task-adaptive gains generated by a small MLP, evaluated on few-shot classification and mini-ImageNet to CUB cross-domain transfer.

Problem

MAML-based few-shot learning generalizes poorly under train/test task distribution shift and needs manual tuning of inner-loop learning rates; existing optimizers lack convergence guarantees for non-convex training.

Approach

Replaces the SGD-momentum inner-loop update with a PID-like rule (proportional, integral from accumulated gradients, derivative from gradient-difference prediction), where the three PID gains are produced per layer by a generated network that takes layer mean, variance and gradients as input, and is meta-learned in the outer loop.

Key results

• LA-PID+MAML with ResNet12 reaches 79.18% on mini-ImageNet 5-way 5-shot vs 69.76% for MAML and 77.96% for ALFA+MAML (1-shot: 63.29% vs 58.37%).
• On mini-ImageNet to CUB cross-domain 5-way 5-shot with 4-CONV, LA-PID+MAML gets 59.73% vs 52.70% for MAML and 58.35% for ALFA+MAML.
• A single LA-PID inner-loop step (66.37%) surpasses MAML with five inner steps (63.11%) on mini-ImageNet 4-CONV 5-way 5-shot.

Measured effects reported in the paper

Surprising findings

• Generated PID gains are non-zero for only a few layers, suggesting the generator does not use the whole network.
• Memory overhead of LA-PID over baseline is modest (e.g., 2944 MB vs 2318 MB for 4-CONV 5w1s), and the LA-PID++ variant is near baseline.

Limitations

• The generator leverages only a few layers' information, as the authors acknowledge, leaving latent cross-network information unused.
• Convergence analysis is a second-order approximation under assumptions (e.g., alpha=1, KP=KI=r) and is not a full non-convex convergence proof.
• Reported 1-shot and 5-shot results are from a limited set of benchmarks and backbones; no comparison with recent non-MAML few-shot methods.

Experimental setup

Datasets: mini-ImageNet tiered-ImageNet FC100 CIFAR-FS CUB-200-2011

Benchmarks: mini-ImageNet tiered-ImageNet FC100 CIFAR-FS CUB-200-2011

Metrics: test accuracy (%)

Baselines: MAML ALFA+MAML L2TT+MAML MAML+Finetune Random Init Baseline++ Baseline

Builds on: MAML SGD with momentum ALFA Ziegler-Nichols tuning

Closest papers by meaning

• A Nested Bi-level Optimization Framework for Robust Few Shot Learning (AAAI 2022 · similarity 0.88)
• How to train your MAML (ICLR 2019 · similarity 0.88)
• On the Global Optimality of Model-Agnostic Meta-Learning (ICML 2020 · similarity 0.87)
• Global Convergence of MAML and Theory-Inspired Neural Architecture Search for Few-Shot Learning (CVPR 2022 · similarity 0.87)
• Meta-Learning with Implicit Gradients (NeurIPS 2019 · similarity 0.87)
• Meta-Learning with Adaptive Hyperparameters (NeurIPS 2020 · similarity 0.87)