Research map / Vision-language and multimodal / Multimodal LLMs
AVerImaTeC: A Dataset for Automatic Verification of Image-Text Claims with Evidence from the Web
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces AVerImaTeC, a dataset of 1,297 real-world image-text claims annotated with web-sourced QA evidence and verdicts, plus a QA-decomposition baseline and a reference-based evidence evaluation for multimodal retrieval.
Problem
Automated verification of real-world image-text claims lacks datasets that are non-synthetic, context-independent, free of temporal leakage, and annotated with evidence capturing verification reasoning.
Approach
Decompose image-text claim verification into QA with open-web evidence annotated under temporal constraints and a two-stage evidence sufficiency check, and evaluate retrieved multimodal evidence with a separated reference-based LLM scorer.
Key results
- Inter-annotator free-marginal kappa of 0.742 on verdicts over 100 re-annotated claims; QA recall 74.7% and precision 67.2%.
- Best few-shot Gemini baseline reaches 0.18 veracity accuracy at evidence threshold 0.2 and 0.12 at 0.4 (hybrid/dynamic), with evidence recall around 0.15–0.25, showing retrieval is hard.
- With ground-truth evidence, Gemini-based baseline reaches 0.82 overall verdict accuracy, while no-search baseline drops to 0.02.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use parallel rather than dynamic question generation | Gemini + Gemini · AVerImaTeC test | Q-Eval recall: 0.33 -> 0.42 | Helped |
| Use parallel rather than dynamic question generation | Gemini + Gemini · AVerImaTeC test | Evidence recall: 0.22 -> 0.15 | Hurt |
| Use hybrid rather than parallel question generation | Gemma + Gemma · AVerImaTeC test | Evidence recall: 0.21 -> 0.25 | Helped |
| Use hybrid rather than parallel question generation | Gemma + Gemma · AVerImaTeC test, λ=0.2 | Conditional justification score: 0.17 -> 0.19 | Helped |
| Use ground-truth evidence instead of disabling external evidence retrieval | Qwen + Qwen-VL · AVerImaTeC test | Overall veracity accuracy: 0.02 -> 0.78 | Helped |
| Use ground-truth evidence instead of disabling external evidence retrieval | Qwen + Qwen-VL · AVerImaTeC test | Justification score: 0.04 -> 0.44 | Helped |
| Use ground-truth evidence instead of disabling external evidence retrieval | Qwen + Qwen-VL · AVerImaTeC test | Evidence recall: 0.06 | Helped |
| Use Ev2R instead of ROUGE-1 to evaluate justifications | Baselines with ground-truth evidence · AVerImaTeC test | Justification evaluation score: 0.50 -> 0.78; 0.44 -> 0.68; 0.43 -> 0.75; 0.49 -> 0.55 | Mixed |
Surprising findings
- Parallel question generation outperformed dynamic and hybrid strategies for verification despite dynamic being designed for multi-hop reasoning.
- Models consistently failed to identify conflicting/cherry-picking claims even with ground-truth evidence.
Limitations
- Relatively small scale (1,297 claims) due to human annotation of real-world claims.
- Reference-based evaluation may underestimate correct but unreferenced model outputs; weak human correlation (Spearman 0.332 for evidence).
- Possible temporal leakage from inaccurate web page date estimates and missing claim dates for about 5% of claims.
Experimental setup
Datasets: AVeriTeC AMMeBa ClaimReview
Benchmarks: AVerImaTeC
Metrics: recall conditional verdict accuracy ROUGE-1 Ev2R Spearman correlation Pearson correlation Randolph's free-marginal kappa Fleiss' kappa
Baselines: Gemini-2.0-flash-001 Qwen2.5-7B-Instruct + Qwen2.5-VL-7B-Instruct Gemma-3-12B Qwen2.5-7B-Instruct + LLaVA-Next-7B
Builds on: AVeriTeC BM25 Ev2R
Closest papers by meaning
- DEFAME: Dynamic Evidence-based FAct-checking with Multimodal Experts (ICML 2025 · similarity 0.88)
- Beyond Text: Fine-Grained Multi-Modal Fact Verification with Hypergraph Transformers (AAAI 2025 · similarity 0.85)
- Heterogeneous Graph Reasoning for Fact Checking over Texts and Tables (AAAI 2024 · similarity 0.85)
- Pioneering Explainable Video Fact-Checking with a New Dataset and Multi-role Multimodal Model Approach (AAAI 2025 · similarity 0.84)
- DeFacto: Counterfactual Thinking with Images for Enforcing Evidence-Grounded and Faithful Reasoning (ICML 2026 · similarity 0.84)
- MRR-FV: Unlocking Complex Fact Verification with Multi-Hop Retrieval and Reasoning (AAAI 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.