Research map / Vision-language and multimodal / CLIP and zero-shot
LAMM: Label Alignment for Multi-Modal Prompt Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LAMM, which replaces class-name tokens with trainable label embeddings for CLIP-based prompt learning and regularizes them with a hierarchical loss (parameter, feature, logits spaces) to improve few-shot, domain generalization, and class-incremental performance.
Problem
Multi-modal prompt tuning of vision-language models focuses on text and visual prompt templates while ignoring the gap between class label representations in the VL model and downstream tasks, which limits few-shot transfer and overfits with few samples.
Approach
Treat each class-name token in the text template as a trainable vector optimized end-to-end with CE loss, and constrain it with a hierarchical loss (weight consolidation in parameter space, cosine alignment in feature space, and knowledge distillation in logits space) so it stays close to the original CLIP label semantics; the method is plug-and-play with CoOp and MaPLe.
Key results
- LAMM improves CLIP by +1.02 to +2.65 points average accuracy across 1-16 shots on 11 datasets, and by +2.57 at 16 shots.
- On 16-shot class-incremental learning (Set1 then Set2), LAMM shows zero degradation on Set 1 while CoOp drops 12.95 and MaPLe drops 5.92 points.
- In domain generalization trained on ImageNet, LAMM averages 61.99 versus MaPLe 61.08 across ImageNet-V2 and ImageNet-Sketch.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add trainable downstream class-label embeddings with hierarchical loss | CLIP · ImageNet domain generalization average | accuracy: 57.90 -> 61.99 | Helped |
| Add trainable downstream class-label embeddings with hierarchical loss | CLIP · ImageNet-V2 | accuracy: 60.83 -> 65.13 | Helped |
| Add trainable downstream class-label embeddings with hierarchical loss | CLIP · ImageNet-Sketch | accuracy: 46.15 -> 48.11 | Helped |
| Add trainable downstream class-label embeddings with hierarchical loss | CLIP · Set 1 after training on Set 2 | accuracy: 69.34 -> 84.14 | Helped |
| Add trainable downstream class-label embeddings with hierarchical loss | CLIP · Set 2 after training | accuracy | Helped |
| Add hierarchical loss (ablation) | LAMM · 11-dataset average, 1-shot | accuracy: 61.39 -> 68.99 | Helped |
| Add hierarchical loss (ablation) | LAMM · 11-dataset average, 16-shot | accuracy: 79.93 -> 81.13 | Helped |
| Add parameter-space WC loss (ablation) | LAMM · 11-dataset average, 16-shot | accuracy: 80.86 -> 81.13 | Helped |
| Add feature-space COS loss (ablation) | LAMM · 11-dataset average, 16-shot | accuracy: 81.08 -> 81.13 | Helped |
| Add logits-space KD loss (ablation) | LAMM · 11-dataset average, 16-shot | accuracy: 80.98 -> 81.13 | Helped |
| Initialize category embeddings from category words rather than randomly (ablation) | LAMM · 11-dataset average, 16-shot | accuracy: 80.71 -> 81.13 | Helped |
| Initialize category embeddings from category words rather than randomly (ablation) | LAMM · 11-dataset average, 1-shot | accuracy: 62.71 -> 68.99 | Helped |
Surprising findings
- At 1-shot, adding LAMM to MaPLe reduces average accuracy (-0.71), with a 5.38-point drop on EuroSAT attributed to overfitting.
- LAMM on vanilla CLIP shows no change on novel classes in base-to-novel testing, since only the trained class embeddings are modified.
Limitations
- Gains at 1-shot are inconsistent, and LAMM can overfit with very few samples (e.g., MaPLe+LAMM on EuroSAT).
- Hyperparameters are fixed across datasets and the authors note that per-dataset tuning could boost performance, so reported results use a single setting.
- Evaluated mainly on image classification with CLIP ViT-B/16 and CNN backbones; applicability to detection or segmentation is not tested.
Experimental setup
Datasets: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 UCF101 DTD EuroSAT
Benchmarks: Caltech101 ImageNet OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 UCF101 DTD EuroSAT ImageNetV2
Metrics: average accuracy accuracy degradation on previous classes
Baselines: zero-shot CLIP CoOp CoCoOp MaPLe
Builds on: CLIP CoOp MaPLe CoCoOp
Closest papers by meaning
- Amend to Alignment: Decoupled Prompt Tuning for Mitigating Spurious Correlation in Vision-Language Models (ICML 2024 · similarity 0.90)
- MmAP: Multi-Modal Alignment Prompt for Cross-Domain Multi-Task Learning (AAAI 2024 · similarity 0.89)
- Learning to Learn Better Visual Prompts (AAAI 2024 · similarity 0.89)
- Compound Text-Guided Prompt Tuning via Image-Adaptive Cues (AAAI 2024 · similarity 0.89)
- COMMA: Co-articulated Multi-Modal Learning (AAAI 2024 · similarity 0.88)
- Weighted Multi-Prompt Learning with Description-free Large Language Model Distillation (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.