Research map / Vision-language and multimodal / CLIP and zero-shot
Exploring the Better Multimodal Synergy Strategy for Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DsRA, a LoRA-based dual-shared adapter mechanism (inter-modal shared coefficients plus intra-modal shared low-rank matrices) to adapt CLIP for few-shot, base-to-new, and domain generalization with fewer parameters than prompt-based methods.
Problem
Prompt-based CLIP adaptation methods such as MaPLe suffer from unidirectional vision-text information flow and large numbers of layer-specific coupling parameters, which hurt few-shot generalization.
Approach
Replace layer-specific prompt coupling functions with a LoRA-style bottleneck whose up/down projection matrices are shared within each encoder, and whose per-layer diagonal coefficient matrices are shared across modalities, so that layer-wise adapters are composed from shared parts to enable mutual vision-text synergy with few parameters.
Key results
- Average 1-shot few-shot accuracy of 73.28% vs MaPLe 69.30% with 0.17M trainable parameters (MaPLe 3.55M).
- Base-to-new harmonic mean 81.10% vs CoPrompt 80.48% and PromptSRC 79.97%.
- Domain generalization average 60.85% vs PromptSRC 60.65% and ALIGN 60.43%; slightly lower on ImageNet-A.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace MaPLe with DsRA | CLIP ViT-B/16 · Few-shot classification (average over 11 datasets), 1-shot | accuracy: +3.98% | Helped |
| Replace ALIGN with DsRA | CLIP ViT-B/16 · Few-shot classification (average over 11 datasets), 1-shot | accuracy: 3.24% | Helped |
| Replace ALIGN with DsRA | CLIP ViT-B/16 · Few-shot classification (average over 11 datasets), 16-shot | accuracy: 1.71% | Helped |
| Use DsRA instead of DePT | CLIP ViT-B/16 · Base-to-new generalization (11 datasets), base classes | accuracy: 85.19% and 77.25% | Helped |
| Use DsRA instead of PromptSRC | CLIP ViT-B/16 · Domain generalization: ImageNet-A | accuracy: 50.19 vs 50.90 | Hurt |
| Replace the shared inter-modal coefficient with modal-specific coefficients (ablation) | DsRA · Few-shot classification (five datasets and average) | accuracy | Mixed |
| Share inter-modal coefficients across modalities (ablation) | DsRA · Few-shot classification (five datasets and average) | accuracy | Helped |
| Place adapters only in MAB rather than both MAB and FNB (ablation) | DsRA · Few-shot classification (average over five datasets) | accuracy: 70.64; 69.50 | Hurt |
| Place adapters only in FNB rather than both MAB and FNB (ablation) | DsRA · Few-shot classification (average over five datasets) | accuracy: 70.64; 69.76 | Hurt |
| Add DsRA to KgCoOp (ablation) | KgCoOp · Base-to-new generalization | HM: from 77.00% to 79.08% | Helped |
| Add DsRA to ProGrad (ablation) | ProGrad · Few-shot classification, 16-shot | accuracy: 2.33% | Helped |
Surprising findings
- Non-shared inter-modal coefficients still beat the baseline, but shared coefficients improve further, suggesting cross-modal sharing helps.
- DsRA is slightly below prior methods on ImageNet-A in domain generalization.
Limitations
- Domain generalization results are comparable rather than uniformly better; lower on ImageNet-A.
- Evaluated only with CLIP ViT-B/16 backbone and image classification tasks.
Experimental setup
Datasets: ImageNet Caltech101 FGVCAircraft Flowers102 Food101 OxfordPets StanfordCars EuroSAT DTD SUN397 UCF101 ImageNetV2
Benchmarks: ImageNet Caltech101 FGVCAircraft Flowers102 Food101 OxfordPets StanfordCars EuroSAT DTD SUN397 UCF101 ImageNetV2
Metrics: accuracy harmonic mean trainable parameters fps
Baselines: CoOp CoCoOp MaPLe ALIGN CoPrompt PromptSRC DePT TCP KgCoOp ProGrad
Builds on: CLIP LoRA MaPLe CoOp CoCoOp
Closest papers by meaning
- LoMo: Local Modality Substitution for Deeper Vision-Language Fusion (NeurIPS 2026 · similarity 0.90)
- MMA: Multi-Modal Adapter for Vision-Language Models (CVPR 2024 · similarity 0.90)
- Modality-Specialized Synergizers for Interleaved Vision-Language Generalists (ICLR 2025 · similarity 0.89)
- Language Models as Black-Box Optimizers for Vision-Language Models (CVPR 2024 · similarity 0.89)
- IPO: Interpretable Prompt Optimization for Vision-Language Models (NeurIPS 2024 · similarity 0.89)
- VladVA: Discriminative Fine-tuning of LVLMs (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.