Research map / Vision-language and multimodal / Multimodal LLMs
CofactGVR: Counterfactual Intervention for Grounded Visual Reasoning
Yan Zhang, Zhijin Qin, Feng Xu, Guiguang Ding, Jungong Han
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CofactGVR, which decomposes grounded visual reasoning into evidence generation followed by counterfactual answer reasoning to provide learning signals for intermediate grounding and reduce shortcut solutions.
Problem
Grounded visual reasoning (GVR) with MLLMs and RL-style fine-tuning lacks effective learning signals for intermediate grounding decisions and is prone to shortcut solutions.
Approach
Explicitly decomposes GVR into Evidence Generation followed by Counterfactual Answer Reasoning, formalized as a Causal Grounding Graph (CGG) in which the generated evidence causally influences the answer; counterfactual interventions on the evidence provide training signals.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add counterfactual intervention bonus (ablation) | CofactGVR · MME | score: 2349.1 -> 2428.5 | Improved |
| add Q-PAS (quantile filter plus prioritized sampling) (ablation) | CofactGVR · MME | score: 2406.2 -> 2428.5 | Improved |
| bound the counterfactual bonus with floor and cap (ablation) | CofactGVR · MME | score: 2341.6 -> 2428.5 | Improved |
| gate the bonus to high-quality factual rollouts (ablation) | CofactGVR · RealworldQA | accuracy: 61.7 -> 66.9 | Improved |
| combine correctness mask and top-k fallback instead of either alone (ablation) | CofactGVR · MME | score: 2302.5 (top-k only) -> 2428.5 | Improved |
| use prioritization exponent alpha 0.5 instead of 0 (ablation) | Q-PAS · MME | score: 2372.7 -> 2428.5 | Improved |
| raise prioritization exponent to 1.0 or above (ablation) | Q-PAS · MME | score: 2428.5 -> 2389.3 | Hurt |
| add filtering before resampling in Q-PAS (ablation) | Q-PAS · RealworldQA | accuracy: 63.8 -> 66.9 | Improved |
| use noise-corrupted random crop instead of no-evidence or plausible-region counterfactual (ablation) | CofactGVR · MME | score: 2306.0 / 2386.4 -> 2428.5 | Improved |
| train with CofactGVR instead of Ground-R1 | Qwen-based MLLM · RefCOCOg val | accuracy: 90.1 -> 89.0 | Hurt |
| train with CofactGVR instead of Ground-R1 | Qwen-based MLLM · RefCOCO+ test-B | accuracy: 78.8 -> 85.3 | Improved |
| preserve predicted evidence versus randomize it at inference | CofactGVR vs Ground-R1 · MME | score gap: 78.7 vs -22.1 | Improved |
Experimental setup
Builds on: GRPO
Closest papers by meaning
- Evidence-RL: Towards Evidence-intensive Visual Reasoning (NeurIPS 2026 · similarity 0.90)
- DeFacto: Counterfactual Thinking with Images for Enforcing Evidence-Grounded and Faithful Reasoning (ICML 2026 · similarity 0.89)
- VGR: Visual Grounded Reasoning (ICLR 2026 · similarity 0.88)
- DeepScan: A Training-Free Framework for Visually Grounded Reasoning in Large Vision-Language Models (CVPR 2026 · similarity 0.88)
- SLVR: Structured Latent Visual Reasoning via Human-like Reasoning Flows (NeurIPS 2026 · similarity 0.88)
- Traceable Evidence Enhanced Visual Grounded Reasoning: Evaluation and Method (ICLR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
