Research map / Image generation / Editing and style transfer
GIM: A Million-scale Benchmark for Generative Image Manipulation Detection and Localization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds GIM, a million-scale generative image manipulation dataset (SAM masks + LLM prompts + Stable Diffusion/GLIDE/DDNM inpainting on ImageNet/VOC) with two benchmark settings, and proposes GIMFormer, a MiT-B2-based dual-branch IMDL model with a learned ShadowTracer trace map.
Problem
Image manipulation detection and localization (IMDL) for generative (AI-based) manipulations lacks large-scale data and benchmarks; existing methods target traditional cheapfakes and small generative datasets.
Approach
Automatically generate local generative manipulations with a SAM-LLM-generator pipeline at large scale, add degradations, and propose GIMFormer combining a learned tampering-trace extractor (ShadowTracer), frequency-spatial block, and multi-window anomaly modeling on a dual-branch transformer.
Key results
- On the mix-generator setting, GIMFormer reaches 70.9% Cls.Acc, 58.6 F1 and 88.2 AUC on GIM-SD, outperforming IML-ViT and SegFormer.
- In the cross-generator setting trained on GIM-SD, GIMFormer gets 67.0% Cls.Acc and 39.3 F1 on GIM-GLIDE and 52.3% Cls.Acc on GIM-DDNM, leading all baselines.
- Ablation on GIM-GLIDE: adding ShadowTracer, MWAM and FSB raises F1 from 56.8 to 77.3 over the SegFormer baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add ShadowTracer to the baseline (ablation) | GIMFormer · GIM-GLIDE | F1: 56.8 -> 67.1 | Helped |
| Add MWAM after ShadowTracer (ablation) | GIMFormer · GIM-GLIDE | F1: 67.1 -> 73.4 | Helped |
| Add FSB after ShadowTracer and MWAM (ablation) | GIMFormer · GIM-GLIDE | F1: 73.4 -> 77.3 | Helped |
| Use pretrained ShadowTracer rather than removing it (ablation) | GIMFormer · GIM-GLIDE, cross-generator | F1: 68.9 -> 78.1 | Helped |
| Use pretrained ShadowTracer rather than removing it (ablation) | GIMFormer · GIM-DDNM, cross-generator | F1: 51.3 -> 58.8 | Helped |
| Increase MWAM from no windows to 1, then 2, then 3 windows (ablation) | GIMFormer · GIM-SD | F1: 58.12 -> 58.34 -> 60.03 -> 61.75 | Helped |
| Change MWAM window sizes to {17,29,41} (ablation) | GIMFormer · GIM-SD | F1: 61.75 -> 61.17 | Hurt |
| Use only the average MWAM branch instead of both branches (ablation) | GIMFormer · GIM-SD | F1: 61.75 -> 60.77 | Hurt |
| Use only the maximum MWAM branch instead of both branches (ablation) | GIMFormer · GIM-SD | F1: 61.75 -> 59.13 | Hurt |
| Increase MWAM from 3 to 4 windows (ablation) | GIMFormer · GIM-SD | F1: 61.75 -> 61.15 | Hurt |
| Use GIMFormer rather than the strongest reported baseline | IMDL methods · GIM-GLIDE, mix-generator | F1: 69.3 -> 77.3 | Helped |
| Use GIMFormer rather than the strongest reported baseline | IMDL methods · GIM-DDNM, cross-generator | F1: 16.8 -> 19.4 | Helped |
Surprising findings
- Blindly increasing manipulated-data volume gave negligible gains once the training set reached about 180K images, so the benchmark was capped at that scale.
- Existing IMDL methods show low pixel-level F1 on the GIM benchmark even when classification accuracy is moderate.
Limitations
- GIM classes overlap with ImageNet and VOC, so novel real-world objects may not be covered.
- Work focuses only on image manipulation; video manipulation is left for future work.
- Pixel-level F1 remains low for all methods, including GIMFormer on the cross-generator settings.
Experimental setup
Datasets: ImageNet-1k PASCAL VOC GIM
Benchmarks: GIM-SD GIM-GLIDE GIM-DDNM GIM-VOC
Metrics: Cls.Acc F1 AUC
Baselines: ManTra-Net MVSS-Net SPAN PSCC-Net ObjectFormer Trufor IML-ViT SegFormer
Builds on: SegFormer SAM Stable Diffusion GLIDE DDNM
Closest papers by meaning
- Towards Modern Image Manipulation Localization: A Large-Scale Dataset and Novel Methods (CVPR 2024 · similarity 0.86)
- Training-Free Image Manipulation Localization Using Diffusion Models (AAAI 2025 · similarity 0.85)
- IMDPrompter: Adapting SAM to Image Manipulation Detection by Cross-View Automated Prompt Learning (ICLR 2025 · similarity 0.84)
- ImgEdit: A Unified Image Editing Dataset and Benchmark (NeurIPS 2025 · similarity 0.84)
- Click2Mask: Local Editing with Dynamic Mask Generation (AAAI 2025 · similarity 0.84)
- GIM: Learning Generalizable Image Matcher From Internet Videos (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.