Research map / Vision-language and multimodal / Multimodal LLMs
Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Identifies CLIP-blind pairs (high CLIP similarity, low DINOv2 similarity), builds the MMVP benchmark that exposes visual failures in GPT-4V and other MLLMs, and proposes Interleaved Mixture-of-Features (I-MoF) combining CLIP and DINOv2 tokens to improve visual grounding.
Problem
MLLMs fail on elementary visual questions, and the visual encoder (CLIP) may be the bottleneck; it is unclear whether CLIP-based vision encoders, including large-scale ones, capture fine-grained visual patterns.
Approach
Mine CLIP-blind image pairs that CLIP embeds similarly but a vision-only SSL model (DINOv2) embeds differently, use them to build a VQA benchmark and a CLIP-model benchmark over nine visual patterns, and fuse CLIP and DINOv2 visual tokens interleaved spatially in the MLLM.
Key results
- Human participants answered 95.7% of MMVP questions correctly, while most MLLMs scored below random guess (25%); GPT-4V was an exception with 38.7% accuracy.
- Interleaved MoF raised LLaVA's MMVP score from 5.5 to 16.7 and LLaVA-1.5's from 24.7 to 28.0 while keeping LLaVA-Bench performance on par.
- Pearson correlation between CLIP and MLLM per-pattern performance was 0.87 for LLaVA-1.5 and 0.71 for InstructBLIP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Interleave CLIP and DINOv2 visual tokens | LLaVA · MMVP | accuracy: 5.5 -> 16.7 (+10.7) | Helped |
| Interleave CLIP and DINOv2 visual tokens | LLaVA · LLaVA benchmark | score: 81.8 -> 82.8 | No clear effect |
| Interleave CLIP and DINOv2 visual tokens | LLaVA-1.5 · MMVP | accuracy: 24.7 -> 28.0 (+3.3) | Helped |
| Interleave CLIP and DINOv2 visual tokens | LLaVA-1.5 · POPE | score: 85.9 -> 86.3 | Helped |
| Increase image resolution from 224² to 336² with I-MoF (ablation) | LLaVA-1.5 + I-MoF · MMVP | accuracy: 28.0 -> 31.3 | Helped |
| Replace DINOv2 with MoCoV3 as the vision-only SSL model (ablation) | LLaVA-1.5 + I-MoF · MMVP | accuracy: 24.7 -> 26.7 | Helped |
| Replace DINOv2 with MAE as the vision-only SSL model (ablation) | LLaVA-1.5 + I-MoF · MMVP | accuracy: 24.7 -> 27.3 | Helped |
| Increase the DINOv2 feature ratio from 0.0 to 0.75 in additive feature mixing (ablation) | LLaVA · MMVP | accuracy: 5.5 -> 18.7 | Helped |
| Increase the DINOv2 feature ratio from 0.0 to 0.75 in additive feature mixing (ablation) | LLaVA · LLaVA benchmark | score: 81.8 -> 75.8 | Hurt |
| Increase the DINOv2 feature ratio beyond 0.75 in additive feature mixing (ablation) | LLaVA · MMVP | accuracy: 18.7 -> 16.5; 13.4 | Hurt |
| Swap the answer options (ablation) | GPT-4V · MMVP | accuracy: 38.7% -> 40.3% | No clear effect |
| Change option labels from letters to numbers (ablation) | GPT-4V · MMVP | accuracy: 38.7% -> 37.3% | No clear effect |
Surprising findings
- Seven of nine visual patterns remained unsolved by large-scale CLIP models, suggesting scaling data and model size alone does not fix these failures.
- Increasing image resolution from 224 to 336 did not improve visual grounding for LLaVA (MMVP 5.5 to 6.0).
Limitations
- Benchmark is small, with 150 CLIP-blind pairs and 300 questions, and questions were manually crafted.
- Additive MoF trades instruction-following for visual grounding, and the interleaved variant doubles visual token handling.
- Closed-source models were accessed through web interfaces at varying times, limiting reproducibility.
Experimental setup
Datasets: ImageNet-1K LAION-Aesthetics CC595k LLaVA 158k DataMix 665k
Benchmarks: MMVP MMVP-VLM POPE LLaVA Bench MMBench GQA TextVQA VQAv2 MM-Vet
Metrics: accuracy pair accuracy Pearson correlation
Baselines: LLaVA-1.5 InstructBLIP MiniGPT-4 GPT-4V Gemini Bard
Builds on: LLaVA LLaVA-1.5 DINOv2 CLIP
Closest papers by meaning
- Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs (NeurIPS 2024 · similarity 0.92)
- Diffusion Feedback Helps CLIP See Better (ICLR 2025 · similarity 0.91)
- Multi-Layer Visual Feature Fusion in Multimodal LLMs: Methods, Analysis, and Best Practices (CVPR 2025 · similarity 0.91)
- Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training (ICLR 2026 · similarity 0.91)
- Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.91)
- Accelerating Pre-training of Multimodal LLMs via Chain-of-Sight (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.