Research map / Vision-language and multimodal / CLIP and zero-shot
Learning to Learn Better Visual Prompts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LoL, which first tunes CoOp prompts on base classes and then applies MAML-style N-way K-shot meta-learning on base-class episodes to improve prompt generalization of CLIP to novel classes.
Problem
CoOp-style prompt tuning of vision-language models overfits to base classes and generalizes poorly to novel classes in base-to-new generalization.
Approach
Treat prompt tuning as learning to learn: after CoOp fine-tuning on base classes, further fine-tune the learnable context vectors with MAML-style episodic N-way K-shot meta-learning over base-class tasks, then apply the learned prompt to unseen classes.
Key results
- With ViT-B/16 at 16 shots, LoL reaches 91.59% new-class accuracy vs 70.90% for CoOp, 69.43% for CoCoOp and 76.52% for KgCoOp (average over 7 datasets).
- Harmonic mean at 16 shots rises to 83.82% for LoL vs 75.72% for KgCoOp on ViT-B/16.
- Applying LoL on top of KgCoOp (LoL-KgCoOp) yields further new-class gains on SUN397, Flowers102 and DTD across shot settings (Table 3).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply meta-learning prompt tuning rather than KgCoOp | CLIP ViT-B/16 · Caltech101, new classes | accuracy: 98.77 -> 99.43 | Helped |
| apply meta-learning prompt tuning rather than KgCoOp | CLIP ViT-B/16 · FGVC Aircraft, new classes | accuracy: 43.53 -> 76.03 | Helped |
| apply meta-learning prompt tuning rather than KgCoOp | CLIP ViT-B/16 · DTD, new classes | accuracy: 66.57 -> 77.04 | Helped |
| apply meta-learning prompt tuning rather than KgCoOp | CLIP ViT-B/16 · Flowers102, new classes | accuracy: 83.50 -> 97.51 | Helped |
| apply meta-learning prompt tuning rather than KgCoOp | CLIP ViT-B/16 · UCF101, new classes | accuracy: 80.77 -> 95.29 | Helped |
| apply meta-learning prompt tuning rather than KgCoOp | CLIP ViT-B/16 · SUN397, new classes | accuracy: 85.57 -> 98.63 | Helped |
| apply meta-learning prompt tuning rather than CoCoOp | CLIP ViT-B/16 · ImageNet, new classes | accuracy: 78.48 -> 97.19 | Helped |
| use word-embedding instead of random context initialization (ablation) | LoL, CLIP ViT-B/16 · FGVC Aircraft, 4 shots | accuracy: 74.10 -> 82.25 | Helped |
| use word-embedding instead of random context initialization (ablation) | LoL, CLIP ViT-B/16 · DTD, 4 shots | accuracy: 76.54 -> 82.51 | Helped |
| use word-embedding instead of random context initialization (ablation) | LoL, CLIP ViT-B/16 · UCF101, 4 shots | accuracy: 93.47 -> 95.06 | Helped |
| use word-embedding instead of random context initialization (ablation) | LoL, CLIP ViT-B/16 · FGVC Aircraft, 8 shots | accuracy: 80.27 -> 83.73 | Helped |
| use word-embedding instead of random context initialization (ablation) | LoL, CLIP ViT-B/16 · DTD, 8 shots | accuracy: 75.76 -> 84.63 | Helped |
Surprising findings
- Base-class accuracy of LoL is identical to CoOp's in the reported tables, so gains come entirely from new-class accuracy.
Limitations
- Base-class accuracy is not improved over CoOp; the method inherits CoOp's base-class performance.
- Results depend on hyperparameters such as query size and context initialization, with no principled selection rule.
- Evaluation covers only 7 image classification datasets with CLIP backbones.
Experimental setup
Datasets: ImageNet Caltech101 Flowers102 FGVC Aircraft SUN397 UCF101 DTD
Benchmarks: ImageNet Caltech101 Flowers102 FGVC Aircraft SUN397 UCF101 DTD
Metrics: accuracy on base classes accuracy on new classes harmonic mean
Baselines: CoOp CoCoOp KgCoOp
Builds on: CoOp MAML KgCoOp CLIP
Closest papers by meaning
- Make Prompts Adaptable: Bayesian Modeling for Vision-Language Prompt Learning with Data-Dependent Prior (AAAI 2024 · similarity 0.93)
- Learning to Prompt with Text Only Supervision for Vision-Language Models (AAAI 2025 · similarity 0.92)
- IPO: Interpretable Prompt Optimization for Vision-Language Models (NeurIPS 2024 · similarity 0.92)
- Amend to Alignment: Decoupled Prompt Tuning for Mitigating Spurious Correlation in Vision-Language Models (ICML 2024 · similarity 0.92)
- Black-Box Test-Time Prompt Tuning for Vision-Language Models (AAAI 2025 · similarity 0.92)
- Skip Tuning: Pre-trained Vision-Language Models are Effective and Efficient Adapters Themselves (CVPR 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.