Research map / Vision-language and multimodal / Multimodal LLMs
ForgerySleuth: Empowering Multimodal Large Language Models for Image Manipulation Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ForgerySleuth, an M-LLM-based image manipulation detector that outputs both tampered-region masks and clue-based reasoning text, trained on a expert-revised ForgeryAnalysis dataset and a GPT-4o/LLaVA-generated pre-training set.
Problem
Multimodal LLMs applied to image manipulation detection produce text-only outputs that suffer from hallucinations and overthinking, and they lack pixel-level localization and low-level forensic cues.
Approach
Combine an M-LLM (LLaVA-7B) with a [SEG] token decoder (LISA-style), a trainable noise-based trace encoder for low-level forensic features, and an attention fusion decoder; train on a Chain-of-Clues dataset ordering high-level doubts before mid/low-level evidence.
Key results
- ForgerySleuth reaches fixed-threshold F1 of 0.562 average and NIST16 AUC 0.898 / IMD20 AUC 0.911, outperforming UnionFormer on challenging datasets.
- On ForgeryAnalysis-Eval, GPT-4 overall score of 9.45 versus 6.96 for GPT-4o, with recall of 92.2%.
- Ablating the trace encoder drops CASIA1 F1 from 0.804 to 0.637 and NIST16 from 0.518 to 0.323.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ForgerySleuth instead of UnionFormer | ForgerySleuth · NIST16 | pixel-level AUC: 0.881 -> 0.898 | Helped |
| Use ForgerySleuth instead of UnionFormer | ForgerySleuth · IMD20 | pixel-level AUC: 0.860 -> 0.911 | Helped |
| Add the trace encoder (ablation) | ForgerySleuth · NIST16 | fixed-threshold F1 (0.5): 0.323 -> 0.518 | Helped |
| Add the trace encoder (ablation) | ForgerySleuth · IMD20 | fixed-threshold F1 (0.5): 0.622 -> 0.710 | Helped |
| Add the fusion mechanism (ablation) | ForgerySleuth · NIST16 | fixed-threshold F1 (0.5): 0.463 -> 0.518 | Helped |
| Add the fusion mechanism (ablation) | ForgerySleuth · IMD20 | fixed-threshold F1 (0.5): 0.649 -> 0.710 | Helped |
| Make the vision encoder trainable instead of frozen (ablation) | ForgerySleuth · NIST16 | fixed-threshold F1 (0.5): 0.518 -> 0.451 | Hurt |
| LoRA fine-tune the vision encoder instead of keeping it frozen (ablation) | ForgerySleuth · NIST16 | fixed-threshold F1 (0.5): 0.518 -> 0.493 | Hurt |
| Add ForgeryAnalysis-SFT after ForgeryAnalysis-PT (ablation) | ForgerySleuth · NIST16 | pixel-level F1: 0.516 -> 0.518 | Helped |
| Add ForgeryAnalysis-SFT after ForgeryAnalysis-PT (ablation) | ForgerySleuth · IMD20 | pixel-level F1: 0.705 -> 0.710 | Helped |
| Add ForgeryAnalysis-SFT after ForgeryAnalysis-PT (ablation) | ForgerySleuth · ForgeryAnalysis-Eval | GPT-4 overall score: 8.71 -> 9.45 | Helped |
| Use ForgeryAnalysis-PT with SFT rather than SFT alone (ablation) | ForgerySleuth · NIST16 | pixel-level F1: 0.191 -> 0.518 | Helped |
Surprising findings
- Fine-tuning or LoRA-tuning the SAM vision encoder reduced performance, so keeping it fully frozen worked best.
- FakeShield's low overall F1 and recall stem largely from missed detections; its performance on true-positive samples is much higher.
Limitations
- Reliance on GPT-4o-generated and GPT-4-judged text introduces possible hallucinations and evaluator bias.
- Pre-training data for ForgeryAnalysis-PT is machine-generated by a 13B LLaVA engine, not fully expert-verified.
Experimental setup
Datasets: ADE20k COCO-Stuff ReasonSeg MIML CASIA2 DEFACTO AutoSplice Columbia Coverage CASIA1 NIST16 IMD20
Benchmarks: Columbia Coverage CASIA1 NIST16 IMD20 COCOGlide ForgeryAnalysis-Eval
Metrics: pixel-level F1 (optimal threshold) pixel-level F1 (fixed threshold 0.5) pixel-level AUC STS GPT-4 score recall human evaluation
Baselines: Mantra-Net SPAN MVSS-Net PSCC-Net CAT-Net2 TruFor UnionFormer ObjectFormer FakeShield GPT-4o Qwen2-VL Qwen
Builds on: LISA LLaVA SAM Chain-of-Thought prompting
Closest papers by meaning
- Unlocking the Capabilities of Large Vision-Language Models for Generalizable and Explainable Deepfake Detection (ICML 2025 · similarity 0.88)
- A Hitchhiker's Guide to Fine-Grained Face Forgery Detection Using Common Sense Reasoning (NeurIPS 2024 · similarity 0.88)
- Towards General Visual-Linguistic Face Forgery Detection (CVPR 2025 · similarity 0.88)
- IDseq: Decoupled and Sequentially Detecting and Grounding Multi-Modal Media Manipulation (AAAI 2025 · similarity 0.87)
- The Coherence Trap: When MLLM-Crafted Narratives Exploit Manipulated Visual Contexts (CVPR 2026 · similarity 0.87)
- CORE: Conflict-Oriented Reasoning for General Multimodal Manipulation Detection (ICML 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.