Research map / Vision-language and multimodal / CLIP and zero-shot
MMA: Multi-Modal Adapter for Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Multi-Modal Adapter (MMA), inserted into the upper transformer layers of CLIP's image and text encoders with a shared projection bridging both branches, guided by a dataset-level recognition analysis of feature discriminability and generalizability.
Problem
Few-shot generalization of pre-trained vision-language models faces a discrimination-generalization dilemma: deciding which representations to preserve versus fine-tune, and how to align text and vision branches during adaptation.
Approach
A multi-modal adapter with separate down/up projections per branch plus a shared projection that passes gradients across modalities, applied only to higher transformer layers (from block 5 in base-to-novel) as chosen by a dataset-level recognition analysis of layer features.
Key results
- Base-to-novel generalization: MMA reaches average HM 79.87 over 11 datasets, above LASP-V (79.48) and MaPLe (78.55) without novel-class information.
- Cross-dataset evaluation: MMA achieves average accuracy 66.61 over 10 datasets, versus MaPLe 66.30.
- Domain generalization: MMA is best on 3 of 4 ImageNet variants (e.g., ImageNet-R 77.32).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MMA instead of the strongest reported baseline | CLIP ViT-B/16 · Base-to-Novel average (11 datasets) | HM: 79.48 -> 79.87 | Helped |
| Use MMA instead of the strongest reported baseline | CLIP ViT-B/16 · Cross-dataset average (10 target datasets) | Accuracy: 66.30 -> 66.61 | Helped |
| Use MMA instead of MaPLe | CLIP ViT-B/16 · ImageNet-Sketch | Accuracy: 49.15 -> 49.13 | Hurt |
| Use MMA instead of MaPLe | CLIP ViT-B/16 · ImageNetV2, ImageNet-A, and ImageNet-R | Accuracy | Helped |
| Add adapters to both branches and a shared projection, rather than use a single-modality adapter or replace MMA with cross-attention (ablation) | MMA · Base-to-Novel average (11 datasets) | HM: 78.02; 77.16; 77.34; 79.87 | Helped |
| Add the shared projection layer (ablation) | MMA without SharedProj · Base-to-Novel average (11 datasets) | HM: 79.20 -> 79.87 | Helped |
| Vary shared-layer dimension from 8 through 128 (ablation) | MMA · Base-to-Novel average (11 datasets) | HM: 79.28; 79.52; 79.87; 79.63; 79.58 | Mixed |
| Vary scaling factor α from 0.0001 through 0.01 (ablation) | MMA · Base-to-Novel average (11 datasets) | HM: 77.44; 78.84; 79.87; 79.36; 78.98 | Mixed |
| Fine-tune the last few transformer layers instead of using MMA (ablation) | CLIP ViT-B/16 · Base-to-Novel average (11 datasets) | HM: 77.28; 78.56; 78.45; 76.59 vs 79.87 | Hurt |
Surprising findings
- Fine-tuning more of the last layers improves base accuracy but hurts novel-class generalization, indicating over-tuning impairs general knowledge.
Limitations
- Adapter insertion layer and scaling factor alpha are chosen by ablation on the evaluation benchmarks, with no separate validation protocol described.
- Results are reported only for ViT-B/16 in the main experiments, limiting evidence for other backbones.
Experimental setup
Datasets: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 DTD EuroSAT UCF101 ImageNetV2
Benchmarks: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 DTD EuroSAT UCF101 ImageNetV2
Metrics: accuracy harmonic mean
Baselines: CLIP CoOp CoCoOp ProDA KgCoOp MaPLe LASP LASP-V RPO PromptSRC
Builds on: CLIP AdaptFormer MaPLe CoCoOp Flamingo-style cross-attention
Closest papers by meaning
- MMRL: Multi-Modal Representation Learning for Vision-Language Models (CVPR 2025 · similarity 0.91)
- AWT: Transferring Vision-Language Models via Augmentation, Weighting, and Transportation (NeurIPS 2024 · similarity 0.90)
- Exploring the Better Multimodal Synergy Strategy for Vision-Language Models (AAAI 2025 · similarity 0.90)
- Modality-Specialized Synergizers for Interleaved Vision-Language Generalists (ICLR 2025 · similarity 0.89)
- Vision-Language Model Selection and Reuse for Downstream Adaptation (ICML 2025 · similarity 0.89)
- Dual Prototype Evolving for Test-Time Generalization of Vision-Language Models (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.