Research map / Detection and segmentation / Open-set and few-shot detection
IMDPrompter: Adapting SAM to Image Manipulation Detection by Cross-View Automated Prompt Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adapts SAM to image manipulation detection and localization by learning automated prompts from four views (RGB, SRM, Bayar, Noiseprint) with optimal prompt selection, cross-view consistency, and feature/prompt fusion, trained on CASIAv2.
Problem
SAM requires manual prompts and single-view (RGB) information generalizes poorly across datasets for image manipulation detection and localization.
Approach
An automated prompt learning paradigm that generates SAM prompts from multiple semantic-agnostic noise views, selects the optimal view's prediction via minimal segmentation loss, enforces cross-view prompt consistency, and fuses features and prompts before SAM's mask decoder.
Key results
- Pixel-level average F1 of 81.36% with best threshold and 59.78% with fixed 0.5 threshold, versus 75.48% and 41.90% for the next best (MVSS-Net).
- Image-level average F1 of 76.23 across CASIA, COVER, Columbia, IMD, versus 53.98 for MVSS-Net; CASIA I-F1 77.3.
- Ablation on COVER: C-F1 rises from 27.0 (RGB baseline) to 66.8 with all components; Noiseprint view alone gives 60.7 C-F1 on COVER.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add CFP (ablation) | IMDPrompter · CASIA | I-F1: 70.6 -> 73.2 | Helped |
| add CPC after CFP (ablation) | IMDPrompter · CASIA | I-F1: 73.2 -> 74.3 | Helped |
| add OPS after CFP and CPC (ablation) | IMDPrompter · CASIA | I-F1: 74.3 -> 75.1 | Helped |
| add PMM after CFP, CPC, and OPS (ablation) | IMDPrompter · CASIA | I-F1: 75.1 -> 77.3 | Helped |
| add CFP (ablation) | IMDPrompter · COVER | C-F1: 27.0 -> 51.8 | Helped |
| add CPC after CFP (ablation) | IMDPrompter · COVER | C-F1: 51.8 -> 57.8 | Helped |
| add OPS after CFP and CPC (ablation) | IMDPrompter · COVER | C-F1: 57.8 -> 63.2 | Helped |
| add PMM after CFP, CPC, and OPS (ablation) | IMDPrompter · COVER | C-F1: 63.2 -> 66.8 | Helped |
| replace RGB-only input with semantic-agnostic Noiseprint view (ablation) | IMDPrompter view combination · COVER | C-F1: 26.00 -> 60.70 | Helped |
| add Bayer view to RGB and Noiseprint views (ablation) | IMDPrompter view combination · COVER | C-F1: 64.30 -> 65.40 | Helped |
| add SRM view to RGB, Noiseprint, and Bayer views (ablation) | IMDPrompter view combination · COVER | C-F1: 65.40 -> 66.80 | Helped |
| replace full SAM with Mobile SAM (ablation) | IMDPrompter · CASIA | C-F1: 76.8 -> 73.1 | Hurt |
Surprising findings
- The RGB view is selected as the optimal prompt only 13.9% of the time, while Noiseprint is selected 46.3%, suggesting semantic features alone are insufficient for guiding SAM.
- Generic SAM-based methods (MedSAM, SAMed, AutoSAM) perform poorly on manipulation detection, indicating that semantic-only SAM adaptation is insufficient.
Limitations
- Cannot detect completely generated images.
- Training requires complete pixel-level supervision.
- Computational cost is high with the SAM-H variant.
Experimental setup
Datasets: CASIAv2 CASIAv1 Columbia COVER IMD2020 NIST16 CocoGlide
Benchmarks: CASIAv1 COVER Columbia IMD2020 NIST16 CocoGlide
Metrics: I-AUC I-F1 P-F1 C-F1 sensitivity specificity pixel F1 best threshold pixel F1 fixed threshold
Baselines: ManTra-Net MVSS-Net TruFor CAT-Net SPAN CR-CNN GSR-Net FCN H-LSTM HP-FCN MedSAM SAMed
Builds on: Segment Anything Model MVSS-Net TruFor CAT-Net
Closest papers by meaning
- AoP-SAM: Automation of Prompts for Efficient Segmentation (AAAI 2025 · similarity 0.87)
- SAM-I2V: Upgrading SAM to Support Promptable Video Segmentation with Less than 0.2% Training Cost (CVPR 2025 · similarity 0.86)
- Endow SAM with Keen Eyes: Temporal-spatial Prompt Learning for Video Camouflaged Object Detection (CVPR 2024 · similarity 0.85)
- Unleashing In-context Learning of Autoregressive Models for Few-shot Image Manipulation (CVPR 2025 · similarity 0.85)
- Stable Segment Anything Model (ICLR 2025 · similarity 0.85)
- Rethinking Interactive Image Segmentation with Low Latency High Quality and Diverse Prompts (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.