Research map / Vision-language and multimodal / Multimodal LLMs
Exploring the Transferability of Visual Prompting for Multimodal Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Transferable Visual Prompting (TVP), a pixel-space visual prompt trained on one MLLM that improves several other MLLMs on recognition, counting, reasoning, and hallucination tasks, using feature consistency alignment and CLIP-based task semantics enrichment.
Problem
Visual prompts trained on one multimodal LLM often fail to transfer to other MLLMs and can degrade their performance, due to cross-model feature corruption; users want shared parameters that improve many MLLMs without per-model fine-tuning.
Approach
Train a single pixel-space visual prompt on one white-box MLLM with an added feature consistency loss (keeping prompted visual features close to clean features) and a CLIP-based task semantics loss (aligning prompted images with task text), so the prompt transfers to unseen MLLMs.
Key results
- TVP trained on InstructBLIP improves average accuracy by +6.69 on CIFAR-10 across six MLLMs, versus +(-1.20) for VP and +3.30 for EVP.
- With ensembling of MiniGPT-4 and InstructBLIP, TVP reaches +31.54 average gain on SVHN and +16.49 on CLEVR counting.
- On CIFAR-10-C corruptions, TVP prompts from MiniGPT-4 give +2.30 average improvement while VP and EVP degrade by -6.62 and -3.87.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TVP instead of EVP for prompts trained on MiniGPT-4 | visual prompting across MLLMs · CIFAR-10 | Avg. Δ: +0.28 -> +3.83 | Helped |
| Use TVP instead of EVP for prompts trained on MiniGPT-4 | visual prompting across MLLMs · ImageNette | Avg. Δ: +0.81 -> +5.80 | Helped |
| Use TVP instead of EVP for prompts trained on MiniGPT-4 | visual prompting across MLLMs · SVHN | Avg. Δ: +17.48 -> +24.72 | Helped |
| Use TVP instead of EVP for prompts trained on MiniGPT-4 | visual prompting across MLLMs · CLEVR | Avg. Δ: +8.64 -> +11.51 | Helped |
| Use TVP instead of EVP for prompts trained on MiniGPT-4 | visual prompting across MLLMs · HM | Avg. Δ: +0.09 -> +1.69 | Helped |
| Use TVP instead of EVP for prompts trained on MiniGPT-4 | visual prompting across MLLMs · POPE | Avg. Δ: +0.90 -> +3.19 | Helped |
| Add FCA to TVP without FCA or TSE (ablation) | visual prompting trained on MiniGPT-4 · CIFAR-10 | Avg. Δ: 0.28 -> 3.82 | Helped |
| Add TSE to TVP without FCA or TSE (ablation) | visual prompting trained on MiniGPT-4 · CIFAR-10 | Avg. Δ: 0.28 -> 1.91 | Helped |
| Add TSE to TVP with FCA (ablation) | visual prompting trained on MiniGPT-4 · CIFAR-10 | Avg. Δ: 3.82 -> 3.83 | No clear effect |
| Set prompt width to 20 rather than 30 (ablation) | TVP trained on MiniGPT-4 · CIFAR-10 | Avg. Δ: +3.83 -> +3.97 | Helped |
| Set prompt width to 80 rather than 30 (ablation) | TVP trained on MiniGPT-4 · CIFAR-10 | Avg. Δ: +3.83 -> -3.03 | Hurt |
| Use TVP rather than VP or EVP under image corruptions | prompts generated on MiniGPT-4 · CIFAR-10-C | Avg. Δ: -6.62% and -3.87% -> 2.30% | Helped |
Surprising findings
- VP and EVP prompts often degrade performance on other MLLMs, even when they improve the model used for training.
- Prompts trained on VisualGLM's counterpart LLM family (GLM) transfer less well, with VisualGLM gaining modestly across methods.
Limitations
- Transfer gains are uneven; VisualGLM (GLM-based LLM) benefits are modest.
- Ensembling increases training cost and storage overhead.
- FCA adds about 32% extra training computation per iteration compared with VP/EVP.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNette SVHN Oxford Pets FGVC-Aircraft Food101 CLEVR Hatefulmemes POPE MSCOCO
Benchmarks: CIFAR-10 ImageNette SVHN CLEVR Hatefulmemes POPE CIFAR-100 Oxford Pets FGVC-Aircraft Food101 CIFAR-10-C ImageNette-C
Metrics: top-1 accuracy AUC average delta
Baselines: VP EVP Zero-Shot
Builds on: visual prompting EVP CLIP transfer attacks
Closest papers by meaning
- ControlMLLM: Training-Free Visual Prompt Learning for Multimodal Large Language Models (NeurIPS 2024 · similarity 0.92)
- VP-Bench: A Comprehensive Benchmark for Visual Prompting in Multimodal Large Language Models (AAAI 2026 · similarity 0.91)
- Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Want (ICLR 2025 · similarity 0.91)
- ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Prompts (CVPR 2024 · similarity 0.91)
- Task Preference Optimization: Improving Multimodal Large Language Models with Vision Task Alignment (CVPR 2025 · similarity 0.90)
- Hugging Visual Prompt and Segmentation Tokens: Consistency Learning for Fine-Grained Visual Understanding in MLLMs (CVPR 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.