Research map / Vision-language and multimodal / Multimodal LLMs
DEFAME: Dynamic Evidence-based FAct-checking with Multimodal Experts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces DEFAME, a zero-shot, six-stage MLLM agent pipeline with web, image, reverse-image and geolocation tools for multimodal claim verification, plus a new post-GPT-4o-cutoff benchmark ClaimReview2024+.
Problem
Automated fact-checking is mostly text-only or lacks retrieval; existing multimodal systems cannot handle both multimodal claims and multimodal evidence, and MLLM parametric knowledge goes stale and leaks benchmark verdicts.
Approach
A modular, training-free MLLM pipeline that iteratively plans tool actions (web search, image search, reverse image search, geolocation), summarizes evidence into a structured multimodal report, develops and judges the verdict, and justifies it, with images kept in context throughout.
Key results
- DEFAME reaches 70.5% accuracy on AVeriTeC dev (vs. 65.6% prior SOTA), 59.2% on MOCHEG, and 83.9% True/False on VERITE.
- On ClaimReview2024+, DEFAME with GPT-4o scores 69.7% versus 35.2% for GPT-4o CoT and 31.4%-35.2% for GPT-4o baselines.
- Human raters found DEFAME's reports significantly better at justifying verdicts than GPT-4o CoT, with no significant difference in coherence.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DEFAME's evidence-retrieval pipeline instead of direct GPT-4o verdict prompting | GPT-4o · AVeriTeC development | accuracy: 62.1 ± 0.4 -> 70.5 ± 0.6 | Helped |
| Use DEFAME's evidence-retrieval pipeline instead of direct GPT-4o verdict prompting | GPT-4o · MOCHEG | accuracy: 53.7 ± 0.4 -> 59.2 ± 0.4 | Helped |
| Use DEFAME's evidence-retrieval pipeline instead of direct GPT-4o verdict prompting | GPT-4o · VERITE true vs. false | accuracy: 78.7 ± 0.8 -> 83.9 ± 0.5 | Helped |
| Use DEFAME's evidence-retrieval pipeline instead of direct GPT-4o verdict prompting | GPT-4o · ClaimReview2024+ | accuracy: 35.2 ± 0.9 -> 69.7 ± 2.5 | Helped |
| Add Geolocation (ablation) | DEFAME · VERITE true vs. false | accuracy: 80.6 -> 83.9 | Helped |
| Add Reverse Image Search (ablation) | DEFAME · VERITE true vs. false | accuracy: 73.7 -> 83.9 | Helped |
| Add Image Search (ablation) | DEFAME · ClaimReview2024+ | accuracy: 63.7 -> 69.7 | Helped |
| Add Web Search (ablation) | DEFAME · MOCHEG | accuracy: 42.0 -> 59.2 | Helped |
| Allow multiple turns and follow-up retrieval (ablation) | DEFAME · MOCHEG | accuracy: 47.7 -> 59.2 | Helped |
| Use dynamic planning instead of a static action schedule (ablation) | DEFAME · ClaimReview2024+ | accuracy: 68.0 -> 69.7 | Helped |
| Add the Develop reasoning stage (ablation) | DEFAME · MOCHEG | accuracy: 57.4 -> 59.2 | Helped |
| Pass retrieved evidence images to later reasoning stages (ablation) | DEFAME · MOCHEG | accuracy: 56.1 -> 59.2 | Helped |
Surprising findings
- Chain-of-thought prompting hurt GPT-4o accuracy on MOCHEG (49.7 vs 53.7) and ClaimReview2024+ (31.4 vs 35.2).
- Removing web search drops MOCHEG accuracy from 59.2 to 42.0, while removing geolocation or reverse search has smaller effects on other datasets.
Limitations
- Accuracy still below human fact-checkers; reliance on search engine credibility and possible leaked content.
- Web scraping instability and formatting sensitivity of open-source MLLMs such as LLaVA-OneVision.
- Hallucination risk not thoroughly analyzed; most failures are label ambiguity between REFUTED and MISLEADING.
Experimental setup
Datasets: AVeriTeC MOCHEG VERITE ClaimReview2024+
Benchmarks: AVeriTeC MOCHEG VERITE ClaimReview2024+
Metrics: accuracy True/False accuracy human coherence rating human completeness rating
Baselines: GPT-4o GPT-4o CoT CFR DeBERTa LVLM4FV MetaSum CHASMA AITR
Builds on: ReAct-style tool use ProgramFC RAGAR GeoCLIP AVeriTeC
Closest papers by meaning
- AVerImaTeC: A Dataset for Automatic Verification of Image-Text Claims with Evidence from the Web (NeurIPS 2025 · similarity 0.88)
- BiDeV: Bilateral Defusing Verification for Complex Claim Fact-Checking (AAAI 2025 · similarity 0.88)
- Pioneering Explainable Video Fact-Checking with a New Dataset and Multi-role Multimodal Model Approach (AAAI 2025 · similarity 0.87)
- Beyond Text: Fine-Grained Multi-Modal Fact Verification with Hypergraph Transformers (AAAI 2025 · similarity 0.86)
- DeFacto: Counterfactual Thinking with Images for Enforcing Evidence-Grounded and Faithful Reasoning (ICML 2026 · similarity 0.85)
- Format Matters: The Robustness of Multimodal LLMs in Reviewing Evidence from Tables and Charts (AAAI 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.