Research map / Label-efficient and robust learning / Zero-/few-shot and meta-learning

Unleashing the Power of Meta-tuning for Few-shot Generalization Through Sparse Interpolated Experts

ICML 2024 · Label-efficient and robust learning › Zero-/few-shot and meta-learning · approach: Meta-learning analysis and benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SMAT, a meta-tuning method that learns sparse interpolated experts (shared dense update masked per expert, merged by a hypernetwork) on a pre-trained ViT to improve both ID and OOD few-shot generalization.

Problem

Meta-tuning of vision foundation models for few-shot learning tends to underperform on out-of-distribution tasks due to meta-overfitting and task interference when all parameters are updated.

Approach

Meta-tune a frozen pre-trained ViT with a set of learned sparse binary masks over a shared dense update, combined per task by hypernetwork-predicted Gumbel-sigmoid expert weights, with a task-specific dense teacher distillation loss and a Lagrangian-constrained target sparsity level; each expert is a sparse interpolation between pre-trained and meta-tuned weights.

Key results

• Without fine-tuning, SMAT reaches 85.36% ID and 67.65% OOD average accuracy vs. 84.23% and 64.10% for PMF on DINO-ViT-Small.
• With LoRA fine-tuning, SMAT+LoRA reaches 85.88% ID and 75.02% OOD average, versus 85.59% and 72.91% for PMF+LoRA.
• With Sup21k ViT-Small, SMAT+full achieves 87.11% ID and 88.05% OOD average, the best among compared methods.

Measured effects reported in the paper

Surprising findings

• Higher expert sparsity improves OOD accuracy at the cost of ID accuracy, revealing a controllable ID-OOD tradeoff.
• PMF meta-tuning with a DINO backbone can underperform simple pre-trained plus fine-tuned baselines on OOD tasks.

Limitations

• Gradient-free expert selection is limited on OOD tasks with large distribution shift such as TrafficSign.
• Gradient-based fine-tuning requires about 4x the GPU memory of gradient-free and about 2x the time of gradient-free adaptation.
• Sparse parameter savings rely on a naive (value, position) storage scheme and are not optimized in memory.

Experimental setup

Datasets: Meta-Dataset ImageNet Omniglot Aircraft CUB DTD Quickdraw Fungi VGG Flower

Benchmarks: Meta-Dataset TrafficSign MSCOCO CIFAR-10 CIFAR-100 MNIST

Metrics: few-shot classification accuracy

Baselines: Pre-trained DINO-ViT-Small PMF SoftMerge (SMEAR) Pre+full PMF+full Pre+LoRA PMF+LoRA SoftMoE

Builds on: PMF SMEAR Prototypical Networks stretched hard concrete distribution LoRA

Closest papers by meaning

• Provable Meta-Learning with Low-Rank Adaptations (NeurIPS 2025 · similarity 0.87)
• Data-Centric Meta-Learning for Robust Few-Shot Generalization (CVPR 2026 · similarity 0.87)
• Learning where to learn: Gradient sparsity in meta and continual learning (NeurIPS 2021 · similarity 0.87)
• Expanding Sparse Tuning for Low Memory Usage (NeurIPS 2024 · similarity 0.87)
• Pushing the Limit of Fine-Tuning for Few-Shot Learning: Where Feature Reusing Meets Cross-Scale Attention (AAAI 2024 · similarity 0.86)
• Patch Rebirth: Fast and Transferable Model Inversion of Vision Transformers (NeurIPS 2026 · similarity 0.86)