Research map / Vision-language and multimodal / Human and embodied modeling
ASAP: Advancing Semantic Alignment Promotes Multi-Modal Manipulation Detecting and Grounding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ASAP, which improves cross-modal semantic alignment (via MLLM captions and LLM explanation texts used in contrastive learning) and a manipulation-guided cross attention plus patch manipulation modeling to detect and ground multimodal media manipulation on the DGM4 dataset.
Problem
Detecting and grounding multi-modal media manipulation (DGM4) requires fine-grained image-text alignment, which prior methods largely neglect; manipulated image-text pairs are unpaired, so standard contrastive alignment is hard to apply.
Approach
Use off-the-shelf MLLM captions and LLM explanation texts as auxiliary bridging text for contrastive alignment (restricted to unmanipulated pairs), and add a manipulation-guided cross-attention loss that uses ground-truth manipulation masks to push attention toward manipulated tokens and patches; all modules are training-only.
Key results
- ASAP reaches 94.38 AUC and 87.71 ACC on DGM4 binary classification, versus 93.19 AUC for HAMMER.
- Average improvement of +2.54 over HAMMER across all DGM4 sub-tasks, outperforming UFAFormer (+1.12).
- Text token grounding Precision of 79.38%, about 4.3 points above HAMMER and 6.03 points above UFAFormer.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add LMA to the HAMMER baseline (ablation) | ASAP · DGM4 | AUC: 93.16 -> 94.28 | Helped |
| Add LMA to the HAMMER baseline (ablation) | ASAP · DGM4 | IoUmean: 76.49 -> 75.90 | Hurt |
| Add LMA to the HAMMER baseline (ablation) | ASAP · DGM4 | text grounding F1: 71.83 -> 76.28 | Helped |
| Add MGCA to LMA (ablation) | ASAP · DGM4 | IoUmean: 75.90 -> 76.95 | Helped |
| Add MGCA to LMA (ablation) | ASAP · DGM4 | text grounding F1: 76.28 -> 76.40 | Helped |
| Add PMM to LMA+MGCA (ablation) | ASAP · DGM4 | IoUmean: 76.95 -> 77.35 | Helped |
| Add PMM to LMA+MGCA (ablation) | ASAP · DGM4 | text grounding F1: 76.40 -> 76.52 | Helped |
| Use both caption and explanation text rather than either alone (ablation) | ASAP · DGM4 | mAP: 87.89 -> 88.53 | Helped |
| Use hard negative patch selection (ablation) | ASAP PMM · DGM4 | IoUmean: 76.92 -> 77.35 | Helped |
| Use ASAP rather than UFAFormer | DGM4 methods · DGM4 | AUC: 93.81 -> 94.38 | Helped |
| Use ASAP rather than UFAFormer | DGM4 methods · DGM4 | mAP: 87.85 -> 88.53 | Helped |
| Use ASAP rather than UFAFormer | DGM4 methods · DGM4 | IoUmean: 78.33 -> 77.35 | Hurt |
Surprising findings
- Adding image features in binary classification slightly reduced image grounding accuracy in the LMA-only ablation.
Limitations
- Gains on binary detection are modest relative to baseline (AUC +1.29 over HAMMER).
- Relies on large off-the-shelf MLLM/LLM to generate auxiliary text, adding training-time cost.
Experimental setup
Datasets: DGM4
Benchmarks: DGM4
Metrics: AUC EER ACC mAP CF1 OF1 IoU mean IoU50 IoU75 Precision
Baselines: CLIP ViLT HAMMER HAMMER++ VIKI UFAFormer
Builds on: HAMMER ViT BERT contrastive learning
Closest papers by meaning
- $\textit{HiMaCon:}$ Discovering Hierarchical Manipulation Concepts from Unlabeled Multi-Modal Data (NeurIPS 2025 · similarity 0.84)
- PERIA: Perceive, Reason, Imagine, Act via Holistic Language and Vision Planning for Manipulation (NeurIPS 2024 · similarity 0.83)
- Beyond Appearance Shifts: Task-Semantic Action Calibration for VLA Models (NeurIPS 2026 · similarity 0.82)
- InfoCon: Concept Discovery with Generative and Discriminative Informativeness (ICLR 2024 · similarity 0.82)
- Generate Subgoal Images before Act: Unlocking the Chain-of-Thought Reasoning in Diffusion Model for Robot Manipulation with Multimodal Prompts (CVPR 2024 · similarity 0.82)
- G3Flow: Generative 3D Semantic Flow for Pose-aware and Generalizable Object Manipulation (CVPR 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.