Research map / Vision-language and multimodal / CLIP and zero-shot
Memory-Space Visual Prompting for Efficient Vision-Language Fine-Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MemVP, which injects projected visual prompts as extra key-value entries into the FFN weights of a language model instead of prepending them to the input, cutting training and inference cost while beating prior PEFT methods on VL tasks.
Problem
Input-space visual prompts for efficient VL model construction greatly increase input length of language models, making PEFT training and inference costly.
Approach
Treat visual prompts as additional key-value memory entries concatenated with FFN weight matrices (memory-space visual prompting), so the input length is unchanged and only a projector and position embeddings are added as trainable parameters.
Key results
- On BART-base and T5-base across VQAv2, GQA and COCO Captions, MemVP reaches average scores of 80.2 and 80.8, above VL-PET (79.8 and 80.4), with language-model FLOPs only 23%-44% of baselines.
- On ScienceQA with LLaMA-13B, MemVP achieves 93.78 average accuracy versus 90.50 for LaVIN (13B) and 90.92 for fully fine-tuned LLaVA-13B.
- Compared with VL-PET on VQAv2, MemVP is 1.7x faster in training and 1.4x faster in inference while using 56% of the training memory.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MemVP instead of VL-PET | BART-base · VQAv2 Karpathy test | VQA Score: 65.3 -> 65.2 | No clear effect |
| use MemVP instead of VL-PET | BART-base · GQA test-dev | VQA Score: 53.9 -> 55.1 | Helped |
| use MemVP instead of VL-PET | T5-base · COCO Captions Karpathy test | CIDEr: 121.2 -> 120.8 | Hurt |
| use MemVP instead of VL-PET | BART-base · VQAv2 Karpathy test | FLOPs (G): 5.0 -> 1.2 | Helped |
| use MemVP instead of VL-PET | T5-base · GQA test-dev | FLOPs (G): 11.3 -> 3.8 | Helped |
| use MemVP instead of LLaVA-LoRA | LLaMA-7B · ScienceQA test | Average accuracy: 90.85 -> 93.07 | Helped |
| use MemVP instead of LLaVA-LoRA | LLaMA-7B · ScienceQA test | Training time (s/batch): 0.49 -> 0.28 | Helped |
| remove visual prompts (ablation) | MemVP with LLaMA-7B · ScienceQA test | Average accuracy: 93.07 -> 85.33 | Hurt |
| replace local visual features with global features (ablation) | MemVP with LLaMA-7B · ScienceQA test | Average accuracy: 93.07 -> 89.01 | Hurt |
| concatenate position embeddings instead of adding them (ablation) | MemVP with LLaMA-7B · ScienceQA test | Average accuracy: 93.07 -> 89.79 | Hurt |
| insert visual prompts in keys only (ablation) | MemVP with LLaMA-7B · ScienceQA test | Average accuracy: 93.07 -> 91.94 | Hurt |
| insert visual prompts in values only (ablation) | MemVP with LLaMA-7B · ScienceQA test | Average accuracy: 93.07 -> 92.78 | Hurt |
Surprising findings
- Inserting visual prompts only as memory keys/values, without extending the input, outperforms input-space prompting that uses 256 visual tokens on ScienceQA with LLaMA-7B.
Limitations
- Speed advantage diminishes for long text generation (e.g., detailed captioning), since input length mainly affects prefill.
- On visual instruction tuning with LLaVA-1.5 data, MemVP-7B underperforms LLaVA-1.5-7B on several benchmarks (e.g., GQA 58.7 vs 62.0).
Experimental setup
Datasets: VQAv2 GQA COCO Captions ScienceQA TVQA How2QA LCS-558K LLaVA-1.5 665K instruction data
Benchmarks: VQAv2 GQA COCO Captions ScienceQA TVQA How2QA SQA MME MMBench POPE
Metrics: VQA score CIDEr accuracy FLOPs training time per batch inference time per batch
Baselines: Full Fine-Tuning Compacter LoRA VL-Adapter VL-PET LLaVA-LoRA LLaMA-Adapter LaVIN LLaVA-1.5-7B MobileVLM-1.7B
Builds on: VL-PET VL-Adapter LLaVA LaVIN LLaMA-Adapter
Closest papers by meaning
- LaViP: Language-Grounded Visual Prompting (AAAI 2024 · similarity 0.91)
- SA²VP: Spatially Aligned-and-Adapted Visual Prompt (AAAI 2024 · similarity 0.90)
- LION: Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge (CVPR 2024 · similarity 0.90)
- Query Efficient Black-Box Visual Prompting with Subspace Learning (CVPR 2025 · similarity 0.90)
- IPO: Interpretable Prompt Optimization for Vision-Language Models (NeurIPS 2024 · similarity 0.90)
- VisMem: Latent Vision Memory Unlocks Potential of Vision-Language Models (CVPR 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.