Research map / Vision-language and multimodal / Multimodal LLMs
ViCrit: A Verifiable Reinforcement Learning Proxy Task for Visual Perception in VLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ViCrit, an RL proxy task where VLMs locate a single synthetic hallucinated span in PixMo-Cap captions, plus ViCrit-Bench (607 samples) for zero-shot hallucination-detection evaluation.
Problem
RL with verifiable rewards has worked for LLM math and code but lacks vision-centric tasks that are both perceptually challenging and unambiguously verifiable, so VLM visual perception has not benefited similarly.
Approach
Inject a single subtle visual hallucination into a 200-word human-written caption and train the VLM with GRPO to output the corrupted span; exact string match gives a binary reward while the task still requires perceiving the whole scene.
Key results
- ViCrit-RL-72B improves Qwen2.5-VL-72B average over eight general VL benchmarks from 59.78 to 63.16 and reduces CHAIRs from 26.4 to 21.0.
- ViCrit-RL-7B raises Qwen2.5-VL-7B on ViCrit-Bench overall from 21.9 to 35.6; ViCrit-RL-72B reaches 43.0 vs. o3 at 47.7.
- ViCrit-RL-72B improves MathVision from 35.2 to 40.1 and VLMsAreBlind from 61.3 to 65.8.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune with ViCrit RL | Qwen2.5-VL-7B-Instruct · CHAIRs | CHAIRs: 28.0 -> 25.2 | Helped |
| Fine-tune with ViCrit RL | Qwen2.5-VL-7B-Instruct · MathVision mini | accuracy: 23.6 -> 25.7 | Helped |
| Fine-tune with ViCrit RL | Qwen2.5-VL-7B-Instruct · Average of 8 general VL benchmarks | Avg.: 50.61 -> 53.01 | Helped |
| Fine-tune with ViCrit RL | Qwen2.5-VL-72B-Instruct · CHAIRs | CHAIRs: 26.4 -> 21.0 | Helped |
| Fine-tune with ViCrit RL | Qwen2.5-VL-72B-Instruct · MathVision mini | accuracy: 35.2 -> 40.1 | Helped |
| Fine-tune with ViCrit RL | Qwen2.5-VL-72B-Instruct · Average of 8 general VL benchmarks | Avg.: 59.78 -> 63.16 | Helped |
| Fine-tune with ViCrit RL | Qwen2.5-VL-7B · ViCrit-Bench | accuracy: 21.9 -> 35.6 | Helped |
| Fine-tune with ViCrit RL | Qwen2.5-VL-72B · ViCrit-Bench | accuracy: 42.4 -> 43.0 | Helped |
| Replace ViCrit RL training with captioning-data SFT (ablation) | Qwen2.5-VL-7B-Instruct · Average of 8 general VL benchmarks | Avg.: 50.61 -> 48.41 | Hurt |
| Replace ViCrit RL training with captioning-data SFT (ablation) | Qwen2.5-VL-72B-Instruct · Average of 8 general VL benchmarks | Avg.: 59.78 -> 60.78 | Helped |
| Replace ViCrit RL training with captioning-data SFT (ablation) | Qwen2.5-VL-7B-Instruct · CHAIRs | CHAIRs: 28.0 -> 25.5 | Helped |
| Fine-tune with ViCrit RL | Qwen2.5-VL-72B-Instruct · ViCrit-Bench Spatial hallucination | accuracy | Hurt |
Surprising findings
- SFT on the same amount of PixMo-Cap captioning data reduces hallucinations but drops the 7B model's general average (50.61 to 48.41).
- Performance on Spatial and Text hallucination categories drops after RL training, attributed to training-data imbalance.
Limitations
- 72B model gains on ViCrit-Bench are marginal, likely because the training set is not challenging enough for it.
- ViCrit-Bench is small (607 samples) and partly built with o3-assisted caption correction and GPT-4-generated hallucinations.
- Training data's natural-image skew; only 10% documents and 7% abstract images.
Experimental setup
Datasets: PixMo-Cap
Benchmarks: CHAIR MMHal MathVista MathVision MathVerse MMMU MMStar MM-Vet BLINK (Blind) CharXiv reasoning ViCrit-Bench
Metrics: CHAIRs CHAIRi MMHal score accuracy exact match accuracy
Baselines: Qwen2.5-VL-7B-Instruct Qwen2.5-VL-72B-Instruct Molmo LLaVA-OneVision InternVL2.5 GPT-4o OpenAI-o1 OpenAI-o3 Gemini-2.5-Pro Qwen2.5-VL-CapSFT
Builds on: GRPO PixMo-Cap Qwen2.5-VL
Closest papers by meaning
- Activating Visual Context and Commonsense Reasoning Through Masked Prediction in VLMs (AAAI 2026 · similarity 0.91)
- VL-RewardBench: A Challenging Benchmark for Vision-Language Generative Reward Models (CVPR 2025 · similarity 0.91)
- Perception-R1: Advancing Multimodal Reasoning Capabilities of MLLMs via Visual Perception Reward (ICLR 2026 · similarity 0.91)
- Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models (ICML 2024 · similarity 0.90)
- From Seeing to Thinking: Decoupling Perception and Reasoning Improves Post-Training of Vision-Language Models (ICML 2026 · similarity 0.90)
- VideoHallu: Evaluating and Mitigating Multi-modal Hallucinations on Synthetic Video Understanding (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.