Research map / Vision-language and multimodal / Multimodal LLMs
Dynamic Multimodal Evaluation with Flexible Complexity by Vision-Language Bootstrapping
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Vision-Language Bootstrapping (VLB), a dynamic multimodal evaluation protocol that generates new VQA variants of SEEDBench, MMBench and MME via image and language edits with an LVLM judge, reducing contamination and exposing LVLM sensitivity.
Problem
Static LVLM benchmarks overlap with pre-training data (data contamination) and have fixed complexity, so they do not reliably measure genuine LVLM capability.
Approach
Dynamically bootstrap existing VQA samples by editing images (adding/removing objects, outpainting) and rephrasing questions (word substitution, sentence rephrasing, relevant/irrelevant context), with difficulty labels and an LVLM judge that checks answer consistency, enabling composable variants of flexible complexity.
Key results
- Image bootstrapping: adding objects (V1) drops accuracy up to 9.36 points on MME for LLaVA-NeXT-Vicuna, while removing objects (V2) slightly improves most models.
- Hardest composition V1+L4 lowers GPT-4o MMBench accuracy from 87.37 to 79.55 and MMvet from 73.94 to 43.66 (vanilla vs hard).
- Human verification reports on average 96% consistency for image strategies and 97% for question strategies across the three datasets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add new objects to images (V1) | GPT-4o · SEEDBench | accuracy (%): 77.59 -> 70.91 | Hurt |
| add new objects to images (V1) | GPT-4o · MMBench | accuracy (%): 87.37 -> 82.28 | Hurt |
| add new objects to images (V1) | GPT-4o · MME | accuracy (%): 77.57 -> 71.04 | Hurt |
| remove existing objects from images (V2) | GPT-4o · SEEDBench | accuracy (%): 77.59 -> 78.11 | Helped |
| remove existing objects from images (V2) | GPT-4o · MMBench | accuracy (%): 87.37 -> 89.06 | Helped |
| remove existing objects from images (V2) | GPT-4o · MME | accuracy (%): 77.57 -> 78.49 | Helped |
| expand original images (V3) | GPT-4o · SEEDBench | accuracy (%): 77.59 -> 73.70 | Hurt |
| expand original images (V3) | GPT-4o · MMBench | accuracy (%): 87.37 -> 85.28 | Hurt |
| expand original images (V3) | GPT-4o · MME | accuracy (%): 77.57 -> 72.30 | Hurt |
| substitute words (L1) or rephrase questions (L2) | GPT-4o · MMBench | accuracy (%): 87.37 -> 86.61; 87.37 -> 85.84 | Hurt |
| add relevant context to questions (L3) | GPT-4o · MMBench | accuracy (%): 87.37 -> 87.60 | Helped |
| add irrelevant context to questions (L4) | GPT-4o · MMBench | accuracy (%): 87.37 -> 81.10 | Hurt |
Surprising findings
- Removing objects (V2) slightly boosts accuracy for most LVLMs, and relevant context (L3) improves accuracy, while other perturbations degrade it.
- Claude3-5V-Sonnet's MMvet score falls from 70.77 to 26.10 under the hardest composition, a much larger drop than other models.
Limitations
- Judge relies on an open LVLM (InternVL-2), so consistency verification is imperfect; before-judge consistency rates were 81-98%.
- Image and question generation depends on closed-source GPT-4V and image editing tools, limiting reproducibility and cost-free use.
- Contamination reduction is argued via dissimilarity rather than measured against actual pre-training corpora of evaluated models.
Experimental setup
Datasets: SEEDBench MMBench MME MMvet LLaVABench LAION-100M CC3M COCO-Caption
Benchmarks: SEEDBench MMBench MME MMvet LLaVABench
Metrics: accuracy contamination rate human verification consistency rate
Baselines: GPT-4o Claude3-5V-Sonnet InternVL-2 XComposer2 DeepSeek-VL Yi-VL-34B
Builds on: MPA DyVal Set-of-Mark prompting PowerPaint
Closest papers by meaning
- Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.91)
- MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI (ICML 2024 · similarity 0.90)
- MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Models (ICLR 2025 · similarity 0.90)
- CoVFT: Context-aware Visual Fine-tuning for Multimodal Large Language Models (CVPR 2026 · similarity 0.90)
- MMBench-Live: A Continuously Evolving Benchmark for Multimodal Models (ICML 2026 · similarity 0.90)
- JourneyBench: A Challenging One-Stop Vision-Language Understanding Benchmark of Generated Images (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.