Research map / Vision-language and multimodal / Cross-modal learning
Unleashing the Potential of Consistency Learning for Detecting and Grounding Multi-Modal Media Manipulation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CSCL, a DGM4 framework with contextual and semantic consistency decoders that build fine-grained consistency matrices supervised by a consistency loss and use forgery-aware reasoning/aggregating to detect and ground multi-modal manipulations.
Problem
Detecting and grounding multi-modal media manipulation (DGM4) in image-text news, where existing methods capture fine-grained local consistency poorly, leading to weak localization of forged content.
Approach
Introduces within-modality contextual consistency (patch/token pairwise similarity) and across-modality semantic consistency (local embeddings vs. global embedding of the other modality), each supervised by a consistency loss, then uses forgery-aware reasoning over the k most reliable and suspicious tokens; text grounding uses a consistency-based threshold filter instead of a separate prediction h
Key results
- CSCL reaches 96.34% AUC, 92.48% mAP, 84.07% IoUm and 76.62% text-grounding F1 on the entire DGM4 dataset, the best among compared methods.
- Compared with Wang et al. [47], CSCL improves IoUm by +3.24, IoU50 by +2.13, IoU75 by +6.78 and text F1 by +3.18 points.
- Ablation shows CSCL on METER backbone improves baseline IoUm from 81.21 to 84.07 and F1 from 73.79 to 76.62.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add contextual and semantic consistency learning | CSCL vs. Wang et al. · DGM4 entire dataset | IoUm: 80.83 -> 84.07 | Helped |
| Add contextual and semantic consistency learning | CSCL vs. Wang et al. · DGM4 entire dataset | F1: 73.44 -> 76.62 | Helped |
| Add contextual and semantic consistency decoders (ablation) | CSCL baseline on DGM4 · DGM4 entire dataset | IoUm: 81.21 -> 84.07 | Helped |
| Add contextual consistency on image and text (ablation) | CSCL · DGM4 entire dataset | IoUm: 81.21 -> 83.70 | Helped |
| Add semantic consistency on image and text (ablation) | CSCL · DGM4 entire dataset | F1: 73.79 -> 76.10 | Helped |
| Add contextual consistency on image and semantic consistency on image (ablation) | CSCL · DGM4 entire dataset | IoUm: 81.21 -> 83.92 | Helped |
| Add contextual consistency on text and semantic consistency on text (ablation) | CSCL · DGM4 entire dataset | F1: 73.79 -> 76.02 | Helped |
| Add a consistency processor (ablation) | Contextual Consistency Decoder · DGM4 entire dataset | IoUm: 81.21 -> 83.37 | Helped |
| Add contextual consistency construction (ablation) | Contextual Consistency Decoder · DGM4 entire dataset | IoUm: 81.21 -> 83.52 | Helped |
| Add forgery-aware reasoning (ablation) | Contextual Consistency Decoder · DGM4 entire dataset | IoUm: 81.21 -> 83.70 | Helped |
| Add semantic consistency construction (ablation) | Semantic Consistency Decoder · DGM4 entire dataset | F1: 73.79 -> 74.58 | Helped |
| Add forgery-aware aggregating (ablation) | Semantic Consistency Decoder · DGM4 entire dataset | F1: 73.79 -> 75.43 | Helped |
Limitations
- Grounding of text manipulation relies on a threshold filter whose value (0.5) is chosen by sweeping on the evaluation data.
- Gains on text grounding precision are not uniform; precision is lower than some baselines' despite higher F1.
Experimental setup
Datasets: DGM4
Benchmarks: DGM4
Metrics: AUC EER ACC mAP CF1 OF1 IoUm IoU50 IoU75 precision
Baselines: TS MAT HAMMER HAMMER++ ViKI UFAFormer Wang et al. (ICASSP'24) CLIP ViLT BETR LUKE
Builds on: HAMMER METER ALBEF UFAFormer
Closest papers by meaning
- IDseq: Decoupled and Sequentially Detecting and Grounding Multi-Modal Media Manipulation (AAAI 2025 · similarity 0.86)
- FakeShield: Explainable Image Forgery Detection and Localization via Multi-modal Large Language Models (ICLR 2025 · similarity 0.86)
- The Coherence Trap: When MLLM-Crafted Narratives Exploit Manipulated Visual Contexts (CVPR 2026 · similarity 0.85)
- Forensics-Bench: A Comprehensive Forgery Detection Benchmark Suite for Large Vision Language Models (CVPR 2025 · similarity 0.85)
- Seeing Through Deception: Uncovering Misleading Creator Intent in Multimodal News with Vision-Language Models (ICLR 2026 · similarity 0.85)
- DAMMFND: Domain-Aware Multimodal Multi-view Fake News Detection (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.