Research map / Optimization / Optimizers
MetaDiff: Meta-Learning with Conditional Diffusion for Few-Shot Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MetaDiff, which treats gradient-descent-based meta-optimization of a few-shot base learner's weights as a conditional diffusion denoising process, avoiding backpropagation through the inner loop.
Problem
Gradient-based meta-learning for few-shot learning requires second-order derivatives through the inner-loop path, causing high memory cost and vanishing gradients that degrade performance.
Approach
Views the inner-loop gradient descent as a reverse diffusion (denoising) process over base-learner weights, and trains a task-conditional UNet (conditioned on support set, predicting noise from initial gradient estimates) with the standard diffusion objective, so no inner-loop differentiation is needed.
Key results
- MetaDiff with ResNet12 reaches 64.99% (1-shot) and 81.21% (5-shot) on miniImageNet, and 72.33% / 86.31% on tieredImageNet.
- With Conv4, MetaDiff achieves 73.18% 5-shot on miniImageNet, above MetaQDA's 72.64%, though MetaQDA is higher on 1-shot (56.41% vs 55.06%).
- GPU memory of MetaDiff stays constant as the number of inner-loop steps grows, while MetaLSTM and ALFA grow linearly.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MetaDiff instead of MetaQDA | Conv4 · miniImagenet 5-way 5-shot | accuracy: 72.64 ± 0.62% -> 73.18 ± 0.64% | Helped |
| Use MetaDiff instead of MetaQDA | Conv4 · tieredImagenet 5-way 5-shot | accuracy: 74.28 ± 0.73% -> 75.46 ± 0.69% | Helped |
| Use MetaDiff instead of MetaQDA | Conv4 · miniImagenet 5-way 1-shot | accuracy: 56.41 ± 0.80% -> 55.06 ± 0.81% | Hurt |
| Use MetaDiff instead of MetaQDA | Conv4 · tieredImagenet 5-way 1-shot | accuracy: 58.11 ± 0.48% -> 57.77 ± 0.90% | Hurt |
| Use MetaDiff instead of MetaQDA | ResNet12 · miniImagenet 5-way 5-shot | accuracy: 80.98 ± 0.75% -> 81.21 ± 0.56% | Helped |
| Use MetaDiff instead of MetaQDA | ResNet12 · tieredImagenet 5-way 1-shot | accuracy: 69.97 ± 0.52% -> 72.33 ± 0.92% | Helped |
| Use MetaDiff instead of MetaQDA | ResNet12 · tieredImagenet 5-way 5-shot | accuracy: 85.51 ± 0.58% -> 86.31 ± 0.62% | Helped |
| Use MetaDiff instead of MetaQDA | ResNet12 · miniImagenet 5-way 1-shot | accuracy: 65.12 ± 0.66% -> 64.99 ± 0.77% | Hurt |
| Replace standard GDA with momentum GDA (ablation) | Prototype classifier on miniImagenet · 5-way 1-shot | accuracy: 60.53 ± 0.86% -> 62.03 ± 0.82% | Helped |
| Replace standard GDA with ALFA (ablation) | Prototype classifier on miniImagenet · 5-way 5-shot | accuracy: 72.43 ± 0.66% -> 80.01 ± 0.61% | Helped |
| Replace L2 loss with cross-entropy loss (ablation) | TCUNet on miniImagenet · 5-way 1-shot | accuracy: 64.99 ± 0.77% -> 62.92 ± 0.79% | Hurt |
| Remove the UNet from TCUNet (ablation) | MetaDiff on miniImagenet · 5-way 1-shot | accuracy: 64.99 ± 0.77% -> 62.72 ± 0.84% | Hurt |
Surprising findings
- MetaQDA with ResNet18 achieves higher 1-shot accuracy on miniImageNet (65.12%) than MetaDiff with ResNet12 (64.99%).
Limitations
- Gains over some baselines are small (around 1-3%) and MetaQDA beats MetaDiff on some settings.
- Inference uses deterministic denoising for T=1000 steps, and convergence is reported around 450 steps.
- Experiments are limited to two ImageNet-derived benchmarks with Conv4 and ResNet12 backbones.
Experimental setup
Datasets: miniImageNet tieredImageNet
Benchmarks: miniImageNet tieredImageNet
Metrics: 5-way 1-shot accuracy 5-way 5-shot accuracy GPU memory convergence
Baselines: iMAML ALFA MeTAL MetaSGD+SiMT GAP ANIL COMLN MetaQDA LEO Meta-Curvature MAML+SiMT ClassifierBaseline
Builds on: MAML ALFA DDPM ANIL MetaLSTM
Closest papers by meaning
- Meta-Learning with Implicit Gradients (NeurIPS 2019 · similarity 0.90)
- SHOT: Suppressing the Hessian along the Optimization Trajectory for Gradient-Based Meta-Learning (NeurIPS 2023 · similarity 0.89)
- MetaNODE: Prototype Optimization as a Neural ODE for Few-Shot Learning (AAAI 2022 · similarity 0.89)
- Bayesian Meta-Learning for the Few-Shot Setting via Deep Kernels (NeurIPS 2020 · similarity 0.88)
- Continuous-Time Meta-Learning with Forward Mode Differentiation (ICLR 2022 · similarity 0.88)
- Memory-Reduced Meta-Learning with Guaranteed Convergence (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.