Research map / Vision-language and multimodal / Multimodal LLMs

VP-Bench: A Comprehensive Benchmark for Visual Prompting in Multimodal Large Language Models

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Benchmarks and analysis of MLLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces VP-Bench, a two-stage benchmark (30K+ visually prompted images across 8 VP shapes and 355 attribute combinations, plus six downstream tasks) that evaluates 28 MLLMs on visual prompt perception and utility.

Problem

Existing benchmarks do not systematically evaluate whether MLLMs can perceive and use visual prompts (bounding boxes, tags, masks, etc.) drawn on images, nor how VP design affects downstream grounded referring tasks.

Approach

A hierarchical taxonomy of 8 VP shapes with fine-grained attributes (color, line width, style) rendered on MS-COCO images for perception QA (existence, enumeration, location, reference), followed by a stage that reuses each model's best-perceived VP on six downstream tasks, with debias questions and VP text descriptions.

Key results

• InternVL3-78B achieves the best Stage 1 average accuracy of 87.97%, slightly below the human baseline of 90.03%.
• Proprietary GPT-4o reaches only 68.80% average VP accuracy, with 57.83% on rough-location queries; open-source models average around 80%.
• Stage 2 downstream accuracy for InternVL3-78B is 79.05%, and removing VP text descriptions drops its Stage 1 average from 87.97% to 81.30%.

Surprising findings

• Open-source models such as InternVL3-78B outperform proprietary GPT-4o on VP perception.
• Simply fine-tuning LLaVA-1.5 on VP data (ViP-LLaVA) gives no consistent downstream gain and degrades at 13B scale.

Limitations

• Stage 2 tasks are limited to a maximum of 200 images per task, and downstream results depend on domain knowledge of models.
• Data is largely drawn from MS-COCO, so the perception evaluation is tied to a single natural-image domain.
• Some reported numbers are inconsistent across text and tables (e.g., averages cited in the text differ from the tables).

Experimental setup

Datasets: MS-COCO SZ-CXR Gleason2019 SD-100 Emotic MapillaryVistas SeeClick PSG

Benchmarks: VP-Bench

Metrics: accuracy

Baselines: GPT-4o Doubao-Seed-1.6 Qwen-VL-Max LLaVA-1.5-7B LLaVA-1.5-13B ViP-LLaVA-7B ViP-LLaVA-13B ViP-Bench SoV

Builds on: ViP-Bench ViP-LLaVA SoM VLMEvalKit

Closest papers by meaning

• ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Prompts (CVPR 2024 · similarity 0.91)
• Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Want (ICLR 2025 · similarity 0.91)
• Exploring the Transferability of Visual Prompting for Multimodal Large Language Models (CVPR 2024 · similarity 0.91)
• PARC: A Quantitative Framework Uncovering the Symmetries within Vision Language Models (CVPR 2025 · similarity 0.90)
• VL-ICL Bench: The Devil in the Details of Multimodal In-Context Learning (ICLR 2025 · similarity 0.90)
• MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI (ICML 2024 · similarity 0.90)