Research map / Vision-language and multimodal / Multimodal LLMs
VCR: A Task for Pixel-Level Complex Reasoning in Vision Language Models via Restoring Occluded Text
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Visual Caption Restoration (VCR), a synthetic Wikipedia-based VQA task where models restore occluded 5-gram captions in images, releasing multilingual datasets and showing SOTA VLMs lag far behind humans.
Problem
Current vision-language models are poorly tested on reasoning over text embedded in images when that text is partially occluded, and OCR or masked-LM style evaluations reduce the task to text processing.
Approach
Synthesize images by stacking a picture above its Wikipedia caption and occluding 5-grams with white rectangles that leave only thin strips of text visible, with adjustable occlusion level to set difficulty; the answer is unique and exact-match scoreable.
Key results
- Human volunteers reach about 91% exact match on English Hard and 98.6% on Chinese Easy, while the best model scores in the paper's tables are far lower on hard settings (e.g., GPT-4o 73.2% EN Hard, Qwen2.5-VL 72B 31.5% ZH Hard).
- On the hidden test set, Claude 3.7 Sonnet reaches 74.0% EM with the image and 58.5% on TEI-only for English, and Qwen2.5-VL-72B reaches 86.5% EM on English.
- LoRA fine-tuning on 16K examples gives mixed gains: CogVLM2 improves across settings, MiniCPM-V2.5 only marginally, and Qwen2-VL-7B stays high with little change.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the visual image (VI) input (ablation) | Qwen2.5-VL 7B · VCR English Easy | Exact match (%): 93.79 -> 94.81 | Helped |
| Add the visual image (VI) input (ablation) | Qwen2.5-VL 7B · VCR English Hard | Exact match (%): 73.83 -> 80.47 | Helped |
| Add the visual image (VI) input (ablation) | Qwen2.5-VL 72B · VCR Chinese Easy | Exact match (%): 72.12 -> 75.82 | Helped |
| Add the visual image (VI) input (ablation) | Qwen2.5-VL 72B · VCR Chinese Hard | Exact match (%): 29.47 -> 31.53 | Helped |
| Fine-tune on the VCR training subset | CogVLM2 · VCR settings | Exact match | Helped |
| Fine-tune on the VCR training subset | MiniCPM-V2.5 · VCR settings | Exact match | No clear effect |
| Fine-tune on the VCR training subset | Qwen2-VL-7B · VCR settings | Exact match | No clear effect |
Surprising findings
- Adding the visual image (VI) context generally hurts performance (negative delta) for most models.
- Models with strong OCR or document understanding, such as InternLM-XComposer2-VL and DocOwl 1.5, perform poorly on VCR.
Limitations
- Fine-tuning on the VCR training set does not consistently yield significant improvements.
- Chinese performance is very low for most models, and the hidden test set is small (100 entities per language).
Experimental setup
Datasets: VCR (Wikipedia WIT image-caption pairs) wikimedia/wit_base
Benchmarks: VCR (EN/ZH, easy/hard) VCR hidden test set
Metrics: exact match Jaccard index
Baselines: GPT-4o Claude 3.5 Sonnet Gemini 1.5 Pro Qwen2-VL Qwen2.5-VL InternVL2.5 CogVLM2 MiniCPM-V2.5 Llama-3.2-Vision
Builds on: LoRA schedule-free AdamW
Closest papers by meaning
- Recoverable Compression: A Multimodal Vision Token Recovery Mechanism Guided by Text Information (AAAI 2025 · similarity 0.89)
- Revisiting the Role of Language Priors in Vision-Language Models (ICML 2024 · similarity 0.88)
- Distilling Vision-Language Models on Millions of Videos (CVPR 2024 · similarity 0.88)
- Cross-Lingual Text-Rich Visual Comprehension: An Information Theory Perspective (AAAI 2025 · similarity 0.88)
- ViCrit: A Verifiable Reinforcement Learning Proxy Task for Visual Perception in VLMs (NeurIPS 2025 · similarity 0.87)
- NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samples (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.