Research map / Image generation / Autoregressive and latent generation
SIDA: Social Media Image Deepfake Detection, Localization and Explanation with Large Multimodal Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SID-Set, a 300K-image social media deepfake dataset (real, FLUX-generated synthetic, and tampered images), and SIDA, a LISA-based large multimodal model that detects, localizes, and explains image forgeries.
Problem
Existing deepfake datasets are mostly facial, use outdated generators, and lack social-media-focused coverage, tampering localization, and textual explanations, so detection, localization and explanation of fake images remain limited.
Approach
Build a large social-media deepfake benchmark with real, fully synthetic and object/part-tampered images, then adapt a LISA large multimodal model with <DET> and <SEG> tokens so one model classifies real/synthetic/tampered, predicts tampered masks, and generates textual explanations.
Key results
- SIDA-7B reaches 93.5% overall accuracy and 93.5 F1 on SID-Set detection, above the compared detectors' best overall numbers in Table 2.
- On tampered-region localization on SID-Set, SIDA-7B achieves 87.3 AUC, 73.9 F1 and 43.8 IoU, versus 82.1/71.3/35.7 for PSCC-Net.
- On DMimage, SIDA-7B reaches 91.8 overall accuracy and 92.4 F1, above AntifakePrompt's 90.6/91.2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the SIDA detection, localization, and explanation framework | SID-Set detection · SID-Set | accuracy: 64.0 -> 93.5 | Helped |
| add the SIDA detection, localization, and explanation framework | SID-Set tampered localization · SID-Set | AUC: 82.1 -> 87.3 | Helped |
| add the SIDA detection, localization, and explanation framework | DMimage · DMimage | accuracy: 90.6 -> 91.8 | Helped |
| add an attention module between detection and segmentation features (ablation) | SIDA · SID-Set tampered localization | AUC: 84.1 -> 87.3 | Helped |
| add an attention module between detection and segmentation features (ablation) | SIDA · SID-Set | accuracy: 90.3 -> 93.5 | Helped |
| replace the attention module with FC layers (ablation) | SIDA · SID-Set tampered localization | IOU: 43.8 -> 38.9 | Hurt |
| set λbce to 4.0 and λdice to 1.0 instead of λbce 2.0 and λdice 0.5 (ablation) | SIDA · SID-Set | F1 Score: 91.01 -> 90.86 | Hurt |
Surprising findings
- Several detectors (e.g., LNP, FreDect) drop in performance on real and synthetic classes after retraining on SID-Set, while their tampered-class scores rise sharply.
Limitations
- Dataset size may be insufficient for the complexity of real social media content.
- Only FLUX is used for synthetic generation, risking data skew toward one generator.
- Some tampered regions are not reliably localized.
Experimental setup
Datasets: SID-Set MagicBrush OpenImages V7 COCO Flickr30k DMimage
Benchmarks: SID-Set DMimage
Metrics: accuracy F1 AUC IoU
Baselines: AntifakePrompt CnnSpot FreDect Fusing Gram-Net UnivFD LGrad LNP MVSS-Net HIFI-Net PSCC-Net LISA-7B-v1
Builds on: LISA LLaVA Language-SAM MagicBrush
Closest papers by meaning
- ODDN: Addressing Unpaired Data Challenges in Open-World Deepfake Detection on Online Social Networks (AAAI 2025 · similarity 0.85)
- WildFake: A Large-Scale and Hierarchical Dataset for AI-Generated Images Detection (AAAI 2025 · similarity 0.84)
- DF40: Toward Next-Generation Deepfake Detection (NeurIPS 2024 · similarity 0.84)
- Towards a Universal Synthetic Video Detector: From Face or Background Manipulations to Fully AI-Generated Content (CVPR 2025 · similarity 0.83)
- Community Forensics: Using Thousands of Generators to Train Fake Image Detectors (CVPR 2025 · similarity 0.83)
- DiffusionFake: Enhancing Generalization in Deepfake Detection via Guided Stable Diffusion (NeurIPS 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.