Research map / Vision-language and multimodal / Multimodal LLMs
ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Prompts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ViP-LLaVA, a multimodal model that overlays arbitrary visual prompts (boxes, arrows, scribbles, masks) directly onto images and fine-tunes LLaVA-1.5 on a new visual-prompt instruction dataset, plus introduces the ViP-Bench benchmark.
Problem
Large vision-language models handle whole-image understanding but struggle with region-specific reference; existing region methods rely on coordinates, positional embeddings, or ROI features and offer no user-friendly arbitrary visual prompting.
Approach
Alpha-blend arbitrary visual markers (rectangles, ellipses, arrows, scribbles, masks, etc.) directly onto the RGB image so CLIP encodes them without any region-specific module, using multi-layer CLIP features and a visual-prompt instruction dataset.
Key results
- ViP-LLaVA-13B reaches 87.91% on Visual7W test versus 85.33% for Shikra-13B, and 88.28% with GPT-4V data.
- ViP-LLaVA-7B achieves 71.31% on PointQA-LookTwice versus 70.30% for Shikra-13B.
- On ViP-Bench, ViP-LLaVA-13B scores 48.3 (synthesized boxes) and 48.2 (human prompts), with only GPT-4V (60.7 and 59.9 overall) ahead among compared models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use multi-layer rather than single-layer CLIP features (ablation) | ViP-LLaVA-Base-7B · Visual7W | accuracy: 82.44 -> 86.04 | Helped |
| use multi-layer rather than single-layer CLIP features (ablation) | ViP-LLaVA-Base-7B · PointQA | accuracy: 70.23 -> 70.86 | Helped |
| use multi-layer rather than single-layer CLIP features (ablation) | ViP-LLaVA-Base-7B · VCR validation | Q → A accuracy: 84.64 -> 87.66 | Helped |
| add original-image tokens to overlaid-image tokens (ablation) | ViP-LLaVA-Base-7B · VCR validation | QA accuracy: 82.47\% -> 81.63\% | No clear effect |
| replace visual prompts with textual coordinates (ablation) | ViP-LLaVA-Base-7B · Visual7W | accuracy: 86.04 -> 86.36 | No clear effect |
| replace visual prompts with textual coordinates (ablation) | ViP-LLaVA-Base-7B · PointQA-LookTwice | accuracy: 70.86 -> 61.4 | Hurt |
| replace visual prompts with textual coordinates (ablation) | ViP-LLaVA-Base-7B · ViP-Bench@Box | score: 45.5 -> 42.6 | Hurt |
| split the overlaid image into source-image and prompt-image inputs (ablation) | ViP-LLaVA-Base-7B · Visual7W | accuracy: 86.04 -> 85.98 | No clear effect |
| split the overlaid image into source-image and prompt-image inputs (ablation) | ViP-LLaVA-Base-7B · PointQA-LookTwice | accuracy: 70.86 -> 70.67 | No clear effect |
| reduce input resolution to 224 pixels and switch to Vicuna v1.1 (ablation) | ViP-LLaVA-Base-7B · Visual7W | accuracy: 86.04 -> 81.80 | Hurt |
| add test-time visual prompt ensembling (ablation) | ViP-LLaVA-Base-7B · Visual7W | accuracy: 86.04 -> 87.44 | Helped |
| add test-time visual prompt ensembling (ablation) | ViP-LLaVA-Base-7B · PointQA-LookTwice | accuracy: 70.86 -> 71.62 | Helped |
Surprising findings
- Overlaying visual prompts on the image gives comparable VCR accuracy to using the original image tokens, suggesting prompt overlays do not obscure visual information.
- Open-source LMMs show large gaps versus GPT-4V on OCR, math and language generation despite decent recognition and knowledge performance.
Limitations
- Open-source models lag GPT-4V on OCR, math, and language generation tasks.
- Region-level instruction data from GPT-4V is small (13k examples).
- Evaluation relies on GPT-4 as judge with in-context examples.
Experimental setup
Datasets: RefCOCOg PointQA-LookTwice Visual Genome Flickr30k Entities Visual Commonsense Reasoning Visual7W LLaVA-1.5 instruction data BLIP 558k captions MM-Vet MMBench COCO
Benchmarks: Visual7W PointQA-LookTwice VCR ViP-Bench RefCOCOg
Metrics: accuracy GPT-4 judged score (0-10) CIDEr METEOR
Baselines: GPT-4V Shikra GPT4RoI LLaVA-1.5 Qwen-VL-Chat InstructBLIP Kosmos-2 GLaMM GRIT
Builds on: LLaVA-1.5 Shikra GPT4RoI Set-of-Mark
Closest papers by meaning
- Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Want (ICLR 2025 · similarity 0.92)
- VP-Bench: A Comprehensive Benchmark for Visual Prompting in Multimodal Large Language Models (AAAI 2026 · similarity 0.91)
- Exploring the Transferability of Visual Prompting for Multimodal Large Language Models (CVPR 2024 · similarity 0.91)
- LION: Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge (CVPR 2024 · similarity 0.90)
- Latent Implicit Visual Reasoning (CVPR 2026 · similarity 0.90)
- Hugging Visual Prompt and Segmentation Tokens: Consistency Learning for Fine-Grained Visual Understanding in MLLMs (CVPR 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.