Research map / Vision-language and multimodal / Multimodal LLMs
Caption This, Reason That: VLMs Caught in the Middle
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Evaluates seven VLMs (incl. GPT-4o) and humans on a procedurally generated Perception-Attention-Memory and composite visual reasoning benchmark, finds a spatial/location CoT bottleneck, and shows self-captioning and LoRA fine-tuning improve performance.
Problem
VLMs lag behind humans on visual tasks such as counting and relational/spatial reasoning, and existing benchmarks do not isolate the low-level cognitive abilities (perception, attention, memory) underlying these failures.
Approach
Builds a cognitive-science-inspired benchmark (PAM plus composite CVR tasks) generated with iWISDM from ShapeNet objects, and uses vision-text decoupling via self-captioning and pre-captioning to show the spatial-reasoning bottleneck is in integrating visual information into CoT rather than in perception or the LLM.
Key results
- GPT-4o nears human accuracy on PAM and exceeds humans on most CVR tasks, but drops to 75.93% on spatial attention versus 98.75% for humans.
- Self-captioning on Qwen2.5-VL-7B raises location-based PAM scores substantially (e.g., Perception Loc 44.33% to 73.00%, +28.67 points), averaging +18.31% on location tasks versus +3.54% on category tasks.
- LoRA fine-tuning on 100k CVR trials improves Qwen2.5-VL-7B on Perception Loc (+34.33) and CVR-Loc-H (+44.33) but MMMU-Pro stays roughly flat (34.36 base vs 34.94 at 10k tasks/100k trials).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace images with model self-captions (ablation) | Qwen2.5-VL-7B · PAM and CVR | accuracy: Perception (Loc) +28.67*; Feature Attention +22.00***; Spatial Attention +18.15***; Memory (Loc) +26.89*** | Helped |
| replace images with ground-truth captions (ablation) | Qwen2.5-VL-7B · PAM and CVR | accuracy: Perception (Loc) +55.00*; Memory (Loc) +47.39***; CVR-Loc-H +32.00* | Helped |
| interleave self-captions with the corresponding images (ablation) | Qwen2.5-VL-7B · PAM | accuracy: Perception (Loc) +10.17; Memory (Loc) +17.07***; Perception (Cat) -6.17; Memory (Cat) -5.27*** | Mixed |
| fine-tune on 100,000 CVR trials (ablation) | Qwen2.5-VL-7B · PAM | accuracy: Perception (Loc) +34.33*; Feature Attention +16.96***; Spatial Attention +17.93***; Memory (Loc) +27.69*** | Helped |
| fine-tune on CVR tasks with 1,000 trials (ablation) | Qwen2.5-VL-7B · PAM | accuracy: Perception (Loc) +17.00; Feature Attention +20.74***; Spatial Attention +24.59*** | Helped |
| fine-tune on CVR tasks with 10,000 trials (ablation) | Qwen2.5-VL-7B · Memory (Cat) | accuracy: -3.20* | Hurt |
| fine-tune on CVR tasks with 10,000 trials (ablation) | Qwen2.5-VL-7B · CVR-Loc-M | accuracy: -8.67 | Hurt |
| fine-tune on CVR tasks with smaller dataset sizes (ablation) | Qwen2.5-VL-7B · MMMU-Pro | score: 34.36 -> 33.70; 34.36 -> 33.83 | Hurt |
| fine-tune on CVR tasks with 100k trials (ablation) | Qwen2.5-VL-7B · MMBench | score: 82.85 -> 83.30 | Helped |
| fine-tune on CVR tasks with 100k trials (ablation) | Qwen2.5-VL-7B · VQAv2 | accuracy (%): 60.07 -> 65.84 | Helped |
| use one-shot prompting (ablation) | Qwen2.5-VL-7B · PAM and CVR | accuracy | No clear effect |
| replace images with self-captions (ablation) | Qwen2.5-VL-72B · CVR | accuracy: CVR-Loc-L +30.67; CVR-Loc-M +28.67; CVR-Cat-M -11.33; CVR-Cat-H -7.83 | Mixed |
Surprising findings
- Self-captioning and pre-captioning gains are much larger for location tasks than category tasks, pointing to a CoT/integration bottleneck rather than perception or the LLM.
- Interleaving images with self-captions (SC-I) hurts category performance relative to baseline, suggesting that adding images alone degrades reasoning.
Limitations
- Stimuli limited to eight ShapeNet object categories with uniform backgrounds, reducing naturalism.
- Fine-tuning restricted to Qwen2.5-VL with up to 100k trials and random complexity distribution.
- Self-captioning roughly doubles inference runtime.
Experimental setup
Datasets: ShapeNet iWISDM-generated PAM tasks iWISDM-generated CVR tasks VQAv2 MMBench MMMU-Pro
Benchmarks: PAM CVR MMBench MMMU-Pro VQAv2
Metrics: accuracy percentage change from base statistical significance
Baselines: GPT-4o-Mini GPT-4o LLaVA-OneVision-7B MiniCPM-V-2.6-8B InternVL2.5-8B Qwen2.5-VL-7B
Builds on: iWISDM Prism framework LoRA
Closest papers by meaning
- From Seeing to Thinking: Decoupling Perception and Reasoning Improves Post-Training of Vision-Language Models (ICML 2026 · similarity 0.92)
- VLMs have Tunnel Vision: Evaluating Nonlocal Visual Reasoning in Leading VLMs (NeurIPS 2025 · similarity 0.92)
- Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective on Focus Areas (ICML 2025 · similarity 0.91)
- Same Task, Different Circuits: Disentangling Modality-Specific Mechanisms in VLMs (NeurIPS 2025 · similarity 0.91)
- Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models (NeurIPS 2024 · similarity 0.91)
- Prism: A Framework for Decoupling and Assessing the Capabilities of VLMs (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.