Research map / Vision-language and multimodal / Multimodal LLMs

DeFacto: Counterfactual Thinking with Images for Enforcing Evidence-Grounded and Faithful Reasoning

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DeFacto, a counterfactual thinking-with-images framework that trains Qwen2.5-VL-7B with GRPO on automatically built positive/counterfactual/random-mask samples (DeFacto-100K) to enforce evidence-answer consistency, and introduces the human-annotated DeFacto-1.5K faithfulness benchmark.

Problem

Thinking-with-images MLLMs often produce answers not supported by correct visual evidence (wrong evidence with correct answer, or correct evidence with wrong answer), so evidence-answer consistency is not guaranteed.

Approach

Combine positive, counterfactual (evidence masked, must answer "Unknown"), and random-masking (irrelevant regions masked) training instances built automatically from question-relevant detections, and optimize with GRPO using answer, format, and region-selection rewards that enforce evidence-grounded abstention.

Key results

• DeFacto improves Qwen2.5-VL-7B on GQA from 53.2 to 63.9 and on VQAv2 from 67.0 to 72.1.
• On DeFacto-1.5K validation, the full model reaches mAP 35.5 and accuracy 60.8 versus 20.9 mAP for GRPO without counterfactual reward.
• On OCRBench, DeFacto scores 871 versus 736 for GPT-4o, the highest reported in its comparison table.

Surprising findings

• SFT without counterfactual data yields mAP 0.0 on DeFacto-1.5K, while adding counterfactual alignment only reaches 2.9 mAP, indicating SFT alone does not produce grounded evidence selection.
• Some baselines underperform the base Qwen2.5-VL substantially on document benchmarks (e.g., GRIT on DocVQA 57.0 vs 92.0), suggesting thinking-with-images training can hurt text-centric tasks.

Limitations

• Region proposals depend on off-the-shelf detectors (RPN, OCR, open-vocabulary detector), which may introduce errors.
• DeFacto-100K is modest relative to large-scale pretraining corpora.
• Evaluation limited to static images; video and temporal reasoning not studied.

Experimental setup

Datasets: VQAv2 ChartQA DocVQA DeFacto-100K

Benchmarks: OKVQA VQAv2 GQA VizWiz ScienceQA VSR DocVQA ChartQA InfoVQA DeepForm Kleister Charity WikiTableQuestions

Metrics: accuracy mAP AP50 AP75 IoU abstention accuracy

Baselines: Qwen2.5-VL ViCrop GRIT DeepEyes Visual-SR1 Chain-of-Focus Pixel Reasoner GPT-4o Gemini-1.5-Pro

Builds on: GRPO Qwen2.5-VL GRIT DeepEyes

Closest papers by meaning

• Evidence-RL: Towards Evidence-intensive Visual Reasoning (NeurIPS 2026 · similarity 0.90)
• Look as You Think: Unifying Reasoning and Visual Evidence Attribution for Verifiable Document RAG via Reinforcement Learning (AAAI 2026 · similarity 0.89)
• CrossCheck-Bench: Diagnosing Compositional Failures in Multimodal Conflict Resolution (AAAI 2026 · similarity 0.88)
• LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models (ICLR 2026 · similarity 0.88)
• SLVR: Structured Latent Visual Reasoning via Human-like Reasoning Flows (NeurIPS 2026 · similarity 0.88)
• CFPO: Counterfactual Policy Optimization for Multimodal Reasoning (ICML 2026 · similarity 0.88)