Research map / Vision-language and multimodal / Multimodal LLMs
SNIFFER: Multimodal Large Language Model for Explainable Out-of-Context Misinformation Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SNIFFER, an explainable MLLM for out-of-context misinformation detection built by two-stage instruction tuning of InstructBLIP with GPT-4-generated OOC instructions plus entity-detection and retrieval tools.
Problem
Out-of-context (OOC) misinformation, where authentic images are paired with false text, is hard to detect and existing detectors give judgments without convincing explanations.
Approach
Adapt a general MLLM (InstructBLIP) to the news domain via two-stage instruction tuning (news-caption alignment, then OOC-specific instructions generated by language-only GPT-4 from caption pairs), and augment it with visual entity detection plus retrieved web evidence checked in an LLM composition step.
Key results
- SNIFFER reaches 88.4% overall accuracy on NewsCLIPpings Merged/Balance, versus 84.7% for CCN and 66.0% for CLIP.
- The original InstructBLIP scores 47.4% accuracy with only 4.6% fake recall; OOC tuning lifts it to 82.5%.
- On a 400-sample subset, SNIFFER scores 86.8% versus 75.5% for GPT-4V.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SNIFFER instead of CCN | NewsCLIPpings Merged/Balance test · NewsCLIPpings Merged/Balance test | All accuracy: 84.7 -> 88.4 | Helped |
| Use SNIFFER instead of GPT-4V | Randomly sampled NewsCLIPpings test set · Randomly sampled NewsCLIPpings test set | All accuracy: 75.5 -> 86.8 | Helped |
| Add news-domain pre-training to InstructBLIP (ablation) | InstructBLIP · NewsCLIPpings Merged/Balance test | All accuracy: 47.4 -> 49.3 | Helped |
| Add OOC task-specific tuning to InstructBLIP (ablation) | InstructBLIP · NewsCLIPpings Merged/Balance test | All accuracy: 47.4 -> 82.5 | Helped |
| Add visual entities to OOC-tuned InstructBLIP (ablation) | InstructBLIP + OOC tuning · NewsCLIPpings Merged/Balance test | All accuracy: 82.5 -> 87.6 | Helped |
| Add news-domain pre-training to OOC-tuned InstructBLIP (ablation) | InstructBLIP + OOC tuning · NewsCLIPpings Merged/Balance test | All accuracy: 82.5 -> 83.1 | Helped |
| Add visual entities to PT- and OOC-tuned InstructBLIP (ablation) | InstructBLIP + PT + OOC tuning · NewsCLIPpings Merged/Balance test | All accuracy: 83.1 -> 88.2 | Helped |
| Add retrieved evidence to PT-, OOC-tuned, and visual-entity SNIFFER (ablation) | SNIFFER without retrieved evidence · NewsCLIPpings Merged/Balance test | All accuracy: 88.2 -> 88.4 | Helped |
| Add retrieved evidence to PT-, OOC-tuned, and visual-entity SNIFFER (ablation) | SNIFFER without retrieved evidence · NewsCLIPpings Merged/Balance test | Real accuracy: 94.0 -> 91.8 | Hurt |
| Use retrieved evidence without PT, OOC tuning, or visual entities (ablation) | InstructBLIP · NewsCLIPpings Merged/Balance test | Fake accuracy: 4.6 -> 92.9 | Helped |
| Use retrieved evidence without PT, OOC tuning, or visual entities (ablation) | InstructBLIP · NewsCLIPpings Merged/Balance test | Real accuracy: 90.3 -> 76.0 | Hurt |
Surprising findings
- The original InstructBLIP accuracy (47.4%) is below random guessing, with almost all fake samples misclassified as real.
- With only 10% of training data, the OOC-tuned variant SNIFFER- matches baselines that do not use external evidence.
Limitations
- Evidence retrieval only covers less than 60% of samples, limiting gains from external checking.
- Retrieved-evidence noise lowers recall on real samples.
- GPT-4V outperforms SNIFFER on fake-sample recall in the comparison (77.0 vs 79.0 is SNIFFER higher; but GPT-4V stronger world knowledge can misjudge real news as fake).
Experimental setup
Datasets: NewsCLIPpings VisualNews News400 TamperedNews
Benchmarks: NewsCLIPpings Merged/Balance News400 TamperedNews GPT-4V comparison subset (400 samples)
Metrics: accuracy (All) accuracy (Fake) accuracy (Real) ROUGE CLIP embedding similarity hit ratio response ratio
Baselines: SAFE EANN VisualBERT CLIP DT-Transformer CCN Neu-Sym detector GPT-4V InstructBLIP
Builds on: InstructBLIP BLIP-2 LLaVA NewsCLIPpings
Closest papers by meaning
- Truth over Tricks: Measuring and Mitigating Shortcut Learning in Misinformation Detection (NeurIPS 2025 · similarity 0.87)
- MMFakeBench: A Mixed-Source Multimodal Misinformation Detection Benchmark for LVLMs (ICLR 2025 · similarity 0.87)
- Fact-R1: Towards Explainable Video Misinformation Detection with Deep Reasoning (NeurIPS 2025 · similarity 0.86)
- Bad Actor, Good Advisor: Exploring the Role of Large Language Models in Fake News Detection (AAAI 2024 · similarity 0.86)
- See Say and Segment: Teaching LMMs to Overcome False Premises (CVPR 2024 · similarity 0.86)
- Seeing Through Deception: Uncovering Misleading Creator Intent in Multimodal News with Vision-Language Models (ICLR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.