Research map / Optimization / Optimizers
Memory-Reduced Meta-Learning with Guaranteed Convergence
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a memory-reduced optimization-based meta-learning algorithm that estimates the hypergradient via a conjugate-gradient Hessian-inverse-vector subroutine using only the last lower-level parameters, with sublinear convergence guarantees.
Problem
Optimization-based meta-learning (MAML, ANIL, iterative differentiation) needs historical lower-level parameters/gradients for hypergradient estimation, causing memory and computational overhead that grows with inner-loop steps.
Approach
Estimate the hypergradient using only the final lower-level parameters, computing the Hessian-inverse-vector product with a stochastic conjugate-gradient subroutine warm-started across outer iterations, avoiding storage of historical parameters/gradients and full Hessian/Jacobian matrices.
Key results
- Test accuracy after 20-step fine-tuning: 64.24% on CIFAR-FS, 48.52% on FC100, 59.58% on miniImageNet, 61.97% on tieredImageNet, outperforming MAML, ANIL, and ITD-BiO.
- Memory consumption is more than 50% lower than MAML, ANIL, and ITD-BiO even at 5 inner-loop iterations, and remains nearly constant as inner iterations grow.
- Theory: sublinear convergence O(1/T)+O(1/|B|) for stochastic meta-learning; exact-solution O(1/T) convergence in deterministic case with complexity O(epsilon^-1).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Algorithm 1's memory-reduced meta-learning method instead of ITD-BiO | Few-shot meta-learning · CIFAR-FS test, 20-step fine-tuning | accuracy (%): 63.69 ± 1.10 -> 64.24 ± 1.41 | Helped |
| Use Algorithm 1's memory-reduced meta-learning method instead of ITD-BiO | Few-shot meta-learning · FC100 test, 20-step fine-tuning | accuracy (%): 47.88 ± 0.74 -> 48.52 ± 1.13 | Helped |
| Use Algorithm 1's memory-reduced meta-learning method instead of ANIL | Few-shot meta-learning · miniImageNet test, 20-step fine-tuning | accuracy (%): 57.80 ± 1.14 -> 59.58 ± 1.08 | Helped |
| Use Algorithm 1's memory-reduced meta-learning method instead of ITD-BiO | Few-shot meta-learning · tieredImageNet test, 20-step fine-tuning | accuracy (%): 60.78 ± 0.10 -> 61.97 ± 0.86 | Helped |
| Use Algorithm 1 instead of ANIL | Few-shot meta-learning · CIFAR-FS | wallclock time (s) to 0.90× Acc_max: 1130.16 -> 832.40 | Helped |
| Use Algorithm 1 instead of ANIL | Few-shot meta-learning · FC100 | wallclock time (s) to 0.90× Acc_max: 879.33 -> 621.67 | Helped |
| Use Algorithm 1 instead of ANIL | Few-shot meta-learning · miniImageNet | wallclock time (s) to 0.90× Acc_max: 1639.85 -> 739.62 | Helped |
| Use Algorithm 1 instead of ANIL | Few-shot meta-learning · tieredImageNet | wallclock time (s) to 0.90× Acc_max: 1417.64 -> 693.46 | Helped |
| Avoid storing historical lower-level parameters/gradients for hypergradient estimation | Algorithm 1 · Few-shot meta-learning; inner-loop iteration K=5 | memory consumption: more than 50% | Helped |
Limitations
- Accuracy gains are small relative to reported standard deviations (e.g., CIFAR-FS 64.24±1.41 vs ANIL 63.52±1.31).
- Convergence bounds rely on hyperparameters K0 and N0 deferred to supplement and on assumptions such as bounded variance and strong convexity of the lower-level problem.
- Experiments are limited to few-shot image classification with a small 4-layer CNN.
Experimental setup
Datasets: CIFAR-FS FC100 miniImageNet tieredImageNet
Benchmarks: CIFAR-FS FC100 miniImageNet tieredImageNet
Metrics: validation accuracy test accuracy after 20-step fine-tuning memory cost wallclock time
Baselines: MAML ANIL ITD-BiO
Builds on: MAML ANIL iMAML stochastic bilevel programming (approximate implicit differentiation)
Closest papers by meaning
- Learning To Learn by Jointly Optimizing Neural Architecture and Weights (CVPR 2022 · similarity 0.90)
- Efficient Variance Reduction for Meta-learning (ICML 2022 · similarity 0.90)
- Submodular Meta-Learning (NeurIPS 2020 · similarity 0.90)
- Efficient Meta Learning via Minibatch Proximal Update (NeurIPS 2019 · similarity 0.89)
- Memory Efficient Online Meta Learning (ICML 2021 · similarity 0.89)
- MetaOptimize: A Framework for Optimizing Step Sizes and Other Meta-parameters (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.