Research map / Vision-language and multimodal / CLIP and zero-shot
Prompt Learning via Meta-Regularization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ProMetaR, which meta-learns a gradient-modulation regularizer and soft prompts for vision-language prompt tuning, with manifold-mixup task augmentation to curb meta-overfitting.
Problem
Prompt tuning of vision-language models like CLIP overfits the small downstream training set, forgetting general knowledge and losing zero-shot generalization to new classes and domains.
Approach
Learn a regularizer that penalizes changes in CLIP embeddings with a meta-learned sigmoid gradient modulation function, optimized with bi-level meta-learning, and augment the meta-validation set via manifold mixup to create virtual tasks and reduce meta-overfitting.
Key results
- Averaged over 11 datasets in base-to-new, ProMetaR reaches 84.39 base, 76.93 new, and 80.49 harmonic mean, improving IVLP by +2.83 H.
- On domain generalization from ImageNet, ProMetaR achieves 60.77 average accuracy over four ImageNet variants, comparable to UNIGRAM (60.80) and above PromptSRC (60.65).
- Plugging ProMetaR into CoOp raises harmonic mean from 71.66 to 76.80, and into VPT from 76.43 to 77.87.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add meta-learning to IVLP (ablation) | IVLP · 11-dataset average, base classes | accuracy: +1.0% | Helped |
| Add meta-learning to IVLP (ablation) | IVLP · 11-dataset average, new classes | accuracy: -0.21% | Hurt |
| Add task augmentation to meta-learning (ablation) | IVLP with meta-learning · 11-dataset average, new classes | accuracy: +2.22% | Helped |
| Add task augmentation to meta-learning (ablation) | IVLP with meta-learning · 11-dataset average | harmonic mean: +1.48% | Helped |
| Add meta-regularization to meta-learning (ablation) | IVLP with meta-learning · 11-dataset average, base classes | accuracy: +0.76% | Helped |
| Add meta-regularization to meta-learning (ablation) | IVLP with meta-learning · 11-dataset average, new classes | accuracy: +1.91% | Helped |
| Add meta-regularization to meta-learning plus task augmentation (ablation) | IVLP with meta-learning and task augmentation · 11-dataset average, new classes | accuracy: +1.56% | Helped |
| Add task augmentation to meta-learning plus meta-regularization (ablation) | IVLP with meta-learning and meta-regularization · 11-dataset average, new classes | accuracy: +1.87% | Helped |
| Add task augmentation and meta-regularization to IVLP (ablation) | IVLP · 11-dataset average, new classes | accuracy: +3.57% | Helped |
| Add ProMetaR to CoOp | CoOp · Base-to-base/base-to-new average | harmonic mean: +5.14% | Helped |
| Add ProMetaR to VPT | VPT · Base-to-base/base-to-new average | harmonic mean: +1.44% | Helped |
| Use ProMetaR instead of IVLP with manually tuned loss regularization | IVLP · 11-dataset average | harmonic mean: 79.62 -> 80.49 | Helped |
Surprising findings
- Meta-learning alone on IVLP slightly reduces new-class accuracy (-0.21%), and task augmentation is needed to make meta-learning help generalization.
Limitations
- Meta-learning on a single task with few samples risks meta-overfitting, only partly addressed by augmentation.
- Gains on some datasets (e.g., Caltech101 base, Food101) are marginal or negative.
- Domain generalization gains are small relative to UNIGRAM, which uses external data.
Experimental setup
Datasets: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 UCF101 DTD EuroSAT
Benchmarks: Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 UCF101 DTD EuroSAT ImageNet ImageNetV2
Metrics: accuracy harmonic mean
Baselines: CLIP CoOp CoCoOp MaPLe RPO PromptSRC UNIGRAM IVLP VPT ProGrad
Builds on: IVLP MAML CoOp ProGrad manifold mixup
Closest papers by meaning
- Learning to Learn Better Visual Prompts (AAAI 2024 · similarity 0.91)
- IPO: Interpretable Prompt Optimization for Vision-Language Models (NeurIPS 2024 · similarity 0.90)
- CoPL: Contextual Prompt Learning for Vision-Language Understanding (AAAI 2024 · similarity 0.90)
- Aggregate-and-Adapt Natural Language Prompts for Downstream Generalization of CLIP (NeurIPS 2024 · similarity 0.90)
- Make Prompts Adaptable: Bayesian Modeling for Vision-Language Prompt Learning with Data-Dependent Prior (AAAI 2024 · similarity 0.89)
- Consistency-guided Prompt Learning for Vision-Language Models (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.