Research map / Vision-language and multimodal / Multimodal LLMs
Once-Tuning-Multiple-Variants: Tuning Once and Expanded as Multiple Vision-Language Model Variants
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Once-Tuning-Multiple-Variants (OTMV), which tunes a VLM once with shared weight tensors and learnable scalar coefficients plus random skip gates so that many smaller variants can be expanded at inference.
Problem
Adapting a vision-language model to multiple deployment scenarios (hardware platforms, latency/throughput needs) currently requires training or fine-tuning several VLMs or compressing one VLM into several compact models, which is costly and complex.
Approach
Borrowing the series expansion idea, most transformer blocks' weight tensors are replaced by a shared key tensor times a learnable scalar OTMV coefficient, and each such tensor is gated with random skip during tuning so that the tuned model can be run with any subset of blocks expanded or skipped at inference.
Key results
- OTMV-tuned BEiT-3 (Large) matches the baseline closely, e.g., VQA test-dev 82.50 vs 82.53 and NLVR2 test-P 89.96 vs 89.95.
- Tuning cost is 6.4 A100 GPU days for OTMV versus 131.8 to 188.8 for compression baselines (about 20-30x less).
- An OTMV-tuned BEiT-3 Large variant with 60% parameter savings reportedly gets a 63% performance improvement on A100.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply OTMV tuning to BEiT-3 (Large) | BEiT-3 (Large) · VQA v2.0, NLVR2, COCO, Flickr30k | task scores: 82.53 -> 82.50; 82.58 -> 82.59; 89.22 -> 89.19; 89.95 -> 89.96; 41.46 -> 41.49; 32.02 -> 32.04; 143.29 -> 143.30; 25.23 | No clear effect |
| Use OTMV tuning instead of EfficientVLM compression | VLM variants · VQA v2.0 | tuning cost: 149.1 (23.3x) -> 6.4 (1x) | Helped |
| Use OTMV tuning instead of DistillVLM compression | VLM variants · VQA v2.0 | tuning cost: 131.8 (20.6x) -> 6.4 (1x) | Helped |
| Use OTMV tuning instead of UPop compression | VLM variants · VQA v2.0 | tuning cost: 188.8 (29.5x) -> 6.4 (1x) | Helped |
| Share multi-head attention weights across modalities using vision weights (ablation) | BEiT-3 (Base) · VQA v2.0 and NLVR2 | accuracy: 78.32 -> 72.77; 84.53 -> 54.96 | Hurt |
| Share multi-head attention weights across modalities using language weights (ablation) | BEiT-3 (Base) · VQA v2.0 and NLVR2 | accuracy: 78.32 -> 53.17; 84.53 -> 51.02 | Hurt |
| Share multi-head attention weights across modalities using vision weights (ablation) | BEiT-3 (Large) · VQA v2.0 and NLVR2 | accuracy: 82.50 -> 78.25; 89.19 -> 57.59 | Hurt |
| Share multi-head attention weights across modalities using language weights (ablation) | BEiT-3 (Large) · VQA v2.0 and NLVR2 | accuracy: 82.50 -> 56.12; 89.19 -> 53.63 | Hurt |
| Use language-modality weights rather than vision-modality weights for cross-modal sharing (ablation) | BEiT-3 attention sharing · VQA v2.0 and NLVR2 | accuracy: 53.17 vs 72.77; 51.02 vs 54.96; 56.12 vs 78.25; 53.63 vs 57.59 | Hurt |
| Share feed-forward weights across modalities rather than within the same modality (ablation) | OTMV-tuned VLM · various benchmarks | task scores: less than 0.4% | No clear effect |
Surprising findings
- Sharing attention weights across vision and language modalities causes large accuracy drops (e.g., VQA falls by over 25 points), while sharing feed-forward weights across modalities costs under 0.4%.
Limitations
- Expanded variants are evaluated only for a limited set of 24 representative variants, and the gap to independently trained models is not fully compared.
- Relies on fine-tuning and per-scenario evaluation on a few VLMs (BLIP, VLMo, BEiT-3) and tasks; generalization to other VLM architectures is not shown.
- The skip probability trades off training difficulty against accuracy drop and must be set by hand.
Experimental setup
Datasets: VQA v2.0 NLVR2 COCO Flickr30K
Benchmarks: VQA v2.0 NLVR2 COCO Flickr30K
Metrics: accuracy (%) I-TR recall@1 T-IR recall@1 BLEU@4 METEOR CIDEr SPICE
Baselines: DistillVLM EfficientVLM UPop MatFormer LoRA OFA network Slimmable network Flextron
Builds on: BEiT-3 VLMo BLIP LoRA UPop
Closest papers by meaning
- Vision-Language Model Selection and Reuse for Downstream Adaptation (ICML 2025 · similarity 0.87)
- Adaptive Parameter Selection for Tuning Vision-Language Models (CVPR 2025 · similarity 0.85)
- Rethinking Model Selection in VLM Through the Lens of Gromov-Wasserstein Distance (CVPR 2026 · similarity 0.85)
- CoVFT: Context-aware Visual Fine-tuning for Multimodal Large Language Models (CVPR 2026 · similarity 0.85)
- OptMerge: Unifying Multimodal LLM Capabilities and Modalities via Model Merging (ICLR 2026 · similarity 0.85)
- Dual Prototype Evolving for Test-Time Generalization of Vision-Language Models (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.