Research map / Vision-language and multimodal / Multimodal LLMs
Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Want
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Draw-and-Understand adds a visual prompt encoder to existing MLLMs (SPHINX-X, LLaVA-Next-LLaMA3) and trains them on a 1.2M-sample multi-domain visual prompt dataset, evaluated on a new MDVP-Bench.
Problem
Existing MLLMs interpret whole images from text, and prior visual-prompt models rely on fixed box references, handle few simultaneous references, need specialized architectures, and often sacrifice image-level perception.
Approach
A general VP-MLLM architecture with a visual prompt encoder that embeds point, box (and noise-simulated free-form) coordinates with learnable markers, plugged into different pre-trained MLLMs; trained in two stages (alignment on detection/segmentation data, then multi-task instruction tuning) with GPT-4V-generated multi-domain data using Set-of-Marks prompting.
Key results
- VP-SPHINX-13B reaches 88.19% GPT-4V ratio on detailed region captioning on RefCOCOg, versus 77.54% for Osprey-7B.
- VP-SPHINX-13B achieves 88.92% Q->A on VCR, above ViP-LLaVA-Base-7B at 87.66%.
- VP-SPHINX-13B scores 45.44 accuracy on COCO Text regional OCR, outperforming ChatSpot at 31.8.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a visual prompt encoder instead of alpha blending (ablation) | SPHINX-13B · LVIS | Semantic Similarity: 56.89 -> 68.74 | Helped |
| Use a visual prompt encoder instead of putting coordinates in text (ablation) | SPHINX-13B · LVIS | Semantic Similarity: 57.74 -> 68.74 | Helped |
| Use two-stage training rather than one-stage training (ablation) | VP-SPHINX-13B · LVIS | Accuracy (Point): 62.94 -> 69.57 | Helped |
| Add GPT-4V-related data to Grounding QA data (ablation) | VP-SPHINX-13B · MDVP Bench Natural | Point score: 52.59 -> 67.73 | Helped |
| Add Grounding QA data to GPT-4V-related data (ablation) | VP-SPHINX-13B · MDVP Bench Natural | Point score: 64.67 -> 67.73 | Helped |
| Use VP-MLLM instead of ChatSpot-7B for box-prompt OCR | COCO-Text · COCO-Text | Accuracy: 31.8 -> 45.44 | Helped |
| Use visual prompting on VP-SPHINX-13B | MDVP Bench · MDVP Bench Natural | Point score: 83.40 -> 92.95 | Helped |
| Use VP-SPHINX-13B rather than SPHINX-X-13B | General image-level understanding · VQA v2 | score: 80.2 -> 78.4 | Hurt |
Surprising findings
- Box prompts sometimes scored lower than point prompts on MDVP-Bench; the authors attribute this to the number of prompts per image.
- VP models show a drop in general image-level benchmarks (e.g., SEED, POPE) relative to original MLLMs, attributed to training on only a subset of open-source instruction data.
Limitations
- Free-form prompts are approximated by enclosing boxes at inference, so they are not truly modeled as shapes.
- Overall image-level performance is somewhat below the original MLLMs on several benchmarks.
- Box-versus-point performance is inconsistent across tasks and prompt counts.
Experimental setup
Datasets: MDVP-Instruct-Data Flickr30K RefCOCO RefCOCO+ RefCOCOg GCG GRIT GeoChat Point-QA Visual7W VCR COCO
Benchmarks: LVIS PACO COCO Text RefCOCOg LLaVA Bench Ferret Bench MDVP-Bench VQA-v2 MME POPE SEED-Bench TextVQA
Metrics: Semantic Similarity Semantic IOU accuracy METEOR CIDEr GPT-4V score ratio GPT-4 relative score
Baselines: Shikra GPT4RoI ChatSpot Osprey Ferret Ferret-v2 Kosmos-2 ViP-LLaVA
Builds on: LLaVA SPHINX-X Set-of-Marks prompting Osprey Ferret
Closest papers by meaning
- ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Prompts (CVPR 2024 · similarity 0.92)
- VP-Bench: A Comprehensive Benchmark for Visual Prompting in Multimodal Large Language Models (AAAI 2026 · similarity 0.91)
- Exploring the Transferability of Visual Prompting for Multimodal Large Language Models (CVPR 2024 · similarity 0.91)
- LION: Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge (CVPR 2024 · similarity 0.91)
- Hugging Visual Prompt and Segmentation Tokens: Consistency Learning for Fine-Grained Visual Understanding in MLLMs (CVPR 2026 · similarity 0.90)
- ControlMLLM: Training-Free Visual Prompt Learning for Multimodal Large Language Models (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.