Research map / Vision-language and multimodal / CLIP and zero-shot
UniAdapter: Unified Parameter-Efficient Transfer Learning for Cross-modal Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UniAdapter, a parameter-efficient adapter framework that unifies unimodal and cross-modal adapters on a frozen BLIP backbone, using a shared down-projection, query-residual adapters, and parameter-free frame-aware attention to match or beat full fine-tuning on six video/image-language benchmarks with about 1-2% tunable parameters.
Problem
Full fine-tuning of large vision-language models is costly in compute and storage across many downstream tasks, and existing adapters handle single modalities or single task types without cross-modal interaction, so a unified parameter-efficient approach for image and video and for retrieval and rea
Approach
Distribute adapters to visual, textual, and cross-modal encoders with a shared down-projection and modality-specific up-projections (partial weight sharing), add a query-residual adapter that preserves language query information in the multimodal encoder, and use a parameter-free frame-aware attention to weight video frames.
Key results
- On MSR-VTT text-to-video retrieval, UniAdapter (r=128, 4.8M tunable) reaches 49.7 R@1, versus 36.7 for CLIP-Prompt.
- UniAdapter (r=512, 19.0M) reaches 52.1 R@1 on DiDeMo, above full fine-tuning at 51.3.
- On MSRVTT-QA, UniAdapter reaches 44.5 validation accuracy versus 42.7 for full fine-tuning.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add query-residual adaptation to the standard Adapter (ablation) | BLIP-base with Adapter · DiDeMo test, text-to-video retrieval | R@1: 47.4 -> 48.7 | Helped |
| add parameter-free frame-aware attention (ablation) | BLIP-base with query-residual Adapter · DiDeMo test, text-to-video retrieval | R@1: 48.7 -> 49.7 | Helped |
| share adapter weights to form UniAdapter (ablation) | BLIP-base with Adapter and PFA · DiDeMo test, text-to-video retrieval | R@1: 49.7 -> 49.0 | Hurt |
| share down-projection weights (ablation) | UniAdapter · MSRVTT-QA valid set | Val Acc: 43.6 -> 44.5 | Helped |
| share only up-projection weights instead of down-projection weights (ablation) | UniAdapter · DiDeMo test, text-to-video retrieval | R@1: 52.1 -> 50.1 | Hurt |
| remove weight sharing (ablation) | UniAdapter · MSRVTT-QA valid set | Val Acc: 44.5 -> 43.6 | Hurt |
| add query-residual mechanism (ablation) | UniAdapter · DiDeMo test, text-to-video retrieval | R@1: 51.6 -> 52.1 | Helped |
| share weights with the textual adapter instead of adding a separate query-residual adapter (ablation) | BLIP-base with query-residual adaptation · DiDeMo test, text-to-video retrieval | R@1: 50.9 -> 51.1 | Helped |
| use adapters in all visual, textual, and cross-modal encoders rather than only the cross-modal encoder (ablation) | BLIP-base with Adapter · DiDeMo test, text-to-video retrieval | R@1: 47.7 -> 49.9 | Helped |
| increase bottleneck dimension from r=128 to r=512 (ablation) | UniAdapter on MSR-VTT · MSR-VTT 1k-A text-to-video retrieval | R@1: 49.7 -> 50.6 | Helped |
| increase inference frames from 8 to 16 (ablation) | UniAdapter (r=512) on DiDeMo · DiDeMo paragraph-to-video retrieval | R@1: 52.1 -> 53.7 | Helped |
| use UniAdapter instead of BLIP full fine-tuning | BLIP-base · MSCOCO 5K text retrieval | R@1: 81.9 -> 80.1 | Hurt |
Surprising findings
- Inserting adapters only into the multimodal encoder outperforms inserting them into visual or textual encoders, and even both unimodal encoders combined.
- Adding adapters to middle layers 5-8 alone gives the best result, above adapters on early or late layers, which the authors read as cross-modal fusion happening mid-network.
Limitations
- Evaluated only on BLIP-base and a single ALBEF extension; generality to other backbones is limited.
- Video tasks use only 8-16 sampled frames per video.
Experimental setup
Datasets: MSR-VTT DiDeMo MSCOCO Flickr30K MSRVTT-QA VQAv2 Visual Genome
Benchmarks: MSR-VTT DiDeMo MSCOCO Flickr30K MSRVTT-QA VQAv2
Metrics: R@1 R@5 R@10 MdR R@Mean accuracy
Baselines: Linear Probe Full fine-tuning Adapter LoRA Sequential Adapter ST Adapter Parallel Adapter MAM Adapter CLIP-Prompt CLIP4Clip OmniVL BLIP
Builds on: Adapter MAM adapter BLIP LGDN
Closest papers by meaning
- Dynamic Adapter with Semantics Disentangling for Cross-lingual Cross-modal Retrieval (AAAI 2025 · similarity 0.89)
- A Wander Through the Multimodal Landscape: Efficient Transfer Learning via Low-rank Sequence Multimodal Adapter (AAAI 2025 · similarity 0.88)
- WikiDO: A New Benchmark Evaluating Cross-Modal Retrieval for Vision-Language Models (NeurIPS 2024 · similarity 0.88)
- Understanding Retrieval-Augmented Task Adaptation for Vision-Language Models (ICML 2024 · similarity 0.87)
- Split to Merge: Unifying Separated Modalities for Unsupervised Domain Adaptation (CVPR 2024 · similarity 0.87)
- MMA: Multi-Modal Adapter for Vision-Language Models (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.