Research map / Vision-language and multimodal / Human and embodied modeling
IDseq: Decoupled and Sequentially Detecting and Grounding Multi-Modal Media Manipulation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes IDseq, an image-driven decoupled sequential framework that splits forgery and content image features (via two disentangled losses and a DETR-like decoder trained without gradient interaction) and uses a Manipulation Indicator Generator to guide text tasks for DGM4.
Problem
Detecting and grounding multi-modal media manipulation (DGM4): classifying manipulation type and localizing manipulated faces and text in image-text pairs, where shared image features cause exclusive competition between forgery and content needs and unverified image features interfere with text task
Approach
Decouple image features into forgery (patch self-inconsistency) and content (fine-grained contrastive) streams, process image tasks with a DIMD whose branches are trained without gradient interaction, and run text tasks sequentially after forged-face localization, using a Gaussian-based manipulation indicator map to highlight manipulated content features.
Key results
- On DGM4, IDseq reaches 94.55 AUC, 11.40 EER and 88.94 ACC for binary classification, outperforming HAMMER++ (93.33 AUC).
- Fine-grained classification mAP of 90.01 and forgery face grounding IoUmean of 83.33, versus 86.41 and 76.46 for HAMMER++.
- Manipulated text grounding F1 of 73.52, compared with 72.59 for HAMMER++ and 72.02 for UFAFormer.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use IDseq instead of UFAFormer | DGM4 · DGM4 test | mAP: 87.85 -> 90.01 | Helped |
| Use IDseq instead of UFAFormer | DGM4 · DGM4 test | IoUmean: 78.33 -> 83.33 | Helped |
| Use IDseq instead of HAMMER++ | DGM4 · DGM4 test | text grounding F1: 72.59 -> 73.52 | Helped |
| Add PSIL (ablation) | IDseq · DGM4 test | IoUmean: 74.56 -> 82.88 | Helped |
| Add FCL (ablation) | IDseq · DGM4 test | IoUmean: 82.62 -> 83.33 | Helped |
| Add FCL (ablation) | IDseq · DGM4 test | text grounding F1: 70.41 -> 73.52 | Helped |
| Use DIMD rather than a coupled decoder (ablation) | IDseq · DGM4 test | IoUmean: 82.62 -> 83.33 | Helped |
| Use DIMD rather than a dual-branch decoder without shared self-attention (ablation) | IDseq · DGM4 test | IoUmean: 49.22 -> 83.33 | Helped |
| Add MIG (ablation) | IDseq · DGM4 test | text grounding F1: 72.14 -> 73.52 | Helped |
| Use the Gaussian fake map instead of a binary fake map (ablation) | MIG in IDseq · DGM4 test | text grounding F1: 72.38 -> 73.52 | Helped |
Surprising findings
- Using the shared self-attention and concatenation in DIMD is crucial; an isolated dual-branch variant drops forgery face grounding IoUmean to 49.22.
Limitations
- Relies on the DGM4 dataset; generalization to other manipulation datasets is not evaluated.
- Text grounding remains the hardest subtask with F1 around 73.5.
Experimental setup
Datasets: DGM4
Benchmarks: DGM4
Metrics: AUC EER ACC mAP CF1 OF1 IoUmean IoU50 IoU75 precision
Baselines: TS MAT BERT LUKE CLIP ViLT HAMMER HAMMER++ UFAFormer
Builds on: HAMMER ALBEF DETR UFAFormer
Closest papers by meaning
- ForgerySleuth: Empowering Multimodal Large Language Models for Image Manipulation Detection (NeurIPS 2025 · similarity 0.87)
- FakeShield: Explainable Image Forgery Detection and Localization via Multi-modal Large Language Models (ICLR 2025 · similarity 0.86)
- Unleashing the Potential of Consistency Learning for Detecting and Grounding Multi-Modal Media Manipulation (CVPR 2025 · similarity 0.86)
- Unleashing In-context Learning of Autoregressive Models for Few-shot Image Manipulation (CVPR 2025 · similarity 0.85)
- Revisiting Tampered Scene Text Detection in the Era of Generative AI (AAAI 2025 · similarity 0.85)
- Can We Model the Artifacts Explicitly? Disentangle Artifacts via Pairwise Edit Relations for Image Manipulation Localization (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.