Research map / Detection and segmentation / CNN features
UnionFormer: Unified-Learning Transformer with Multi-View Representation for Image Manipulation Detection and Localization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UnionFormer, a unified-learning transformer that fuses RGB, noise, and object-consistency views via a BSFI-Net CNN-Transformer encoder for joint image manipulation detection and localization.
Problem
Image manipulation detection and localization need generalizable features that capture low-level noise artifacts, boundary inconsistencies, and object-level inconsistencies across diverse tampering types, including diffusion-based edits.
Approach
Combines a dual-stream (RGB and constrained-CNN noise view) BSFI-Net encoder with DCT-based edge enhancement and a DETR-style transformer that jointly models object consistency, detection, and mask-vector localization under a unified multi-scale supervision, with contrastive loss linking the two streams.
Key results
- Pixel-level AUC average 0.929 across localization datasets vs 0.927 for TruFor (best baseline), with best AUC on all datasets.
- Image-level detection AUC average 0.864 vs 0.839 for TruFor; balanced accuracy average 0.776 vs 0.756.
- Ablation on NIST16: RGB-only baseline AUC 0.724 rising to 0.881 with noise view, object view, and contrastive supervision.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the noise view to the RGB view (ablation) | UnionFormer · CASIA v1 | pixel-level AUC: 0.778 -> 0.865 | Helped |
| add the noise view to the RGB view (ablation) | UnionFormer · NIST 16 | pixel-level AUC: 0.724 -> 0.807 | Helped |
| add object view representation (ablation) | RGB + Noise Views · CASIA v1 | pixel-level AUC: 0.865 -> 0.950 | Helped |
| add object view representation (ablation) | RGB + Noise Views · NIST 16 | pixel-level AUC: 0.807 -> 0.853 | Helped |
| add contrastive supervision (ablation) | RGB + Noise + Object Views · CASIA v1 | pixel-level AUC: 0.950 -> 0.972 | Helped |
| add contrastive supervision (ablation) | RGB + Noise + Object Views · NIST 16 | pixel-level AUC: 0.853 -> 0.881 | Helped |
| replace ResNet with BSFI-Net (ablation) | UnionFormer · CASIA v1 | pixel-level AUC: 0.895 -> 0.972 | Helped |
| replace ResNet with BSFI-Net (ablation) | UnionFormer · NIST 16 | pixel-level AUC: 0.826 -> 0.881 | Helped |
| remove BOB (ablation) | BSFI-Net · NIST 16 | localization AUC: decrease by 4.8% | Hurt |
| remove FCU (ablation) | BSFI-Net · NIST 16 | localization AUC: decrease by 6.3% | Hurt |
Limitations
- Relies on pre-training BSFI-Net and staged training, increasing pipeline complexity
- Some baseline values taken from literature rather than re-run, which may affect fairness
- Object view depends on RPN proposals, and the mask vector PCA encoding limits spatial detail
Experimental setup
Datasets: CASIA v2 Fantastic Reality Tampered COCO Tampered RAISE COCO 2017 RAISE CASIA v1 Columbia Coverage NIST16 IMD20 CocoGlide
Benchmarks: CASIA v1 Columbia Coverage NIST16 IMD20 CocoGlide BDNIE
Metrics: pixel-level AUC pixel-level F1 image-level AUC balanced accuracy
Baselines: ManTra-Net SPAN MVSS-Net PSCC-Net CAT-Net v2 TruFor ObjectFormer
Builds on: DETR SOLQ Conformer (BSFI-Net inspiration) constrained convolutional layer TruFor
Closest papers by meaning
- Towards Modern Image Manipulation Localization: A Large-Scale Dataset and Novel Methods (CVPR 2024 · similarity 0.86)
- Learning Discriminative Noise Guidance for Image Forgery Detection and Localization (AAAI 2024 · similarity 0.85)
- Training-Free Image Manipulation Localization Using Diffusion Models (AAAI 2025 · similarity 0.85)
- IMDPrompter: Adapting SAM to Image Manipulation Detection by Cross-View Automated Prompt Learning (ICLR 2025 · similarity 0.84)
- Can We Model the Artifacts Explicitly? Disentangle Artifacts via Pairwise Edit Relations for Image Manipulation Localization (NeurIPS 2026 · similarity 0.84)
- Mesoscopic Insights: Orchestrating Multi-Scale & Hybrid Architecture for Image Manipulation Localization (AAAI 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.