Research map / Image generation / Editing and style transfer

Can We Model the Artifacts Explicitly? Disentangle Artifacts via Pairwise Edit Relations for Image Manipulation Localization

NeurIPS 2026 · Image generation › Editing and style transfer · approach: Localized and attribute-specific editing

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Reinterprets image manipulation localization as latent-variable learning and proposes Pairwise Artifacts Learning (PAL), a pre-finetuning stage that disentangles artifacts via edit-related vs. non-related image pairs on the new EditGroup-45K dataset, then uses the PAL encoder to initialize standard IML models.

Problem

Image manipulation localization (IML) is usually trained end-to-end as P(y|x), which implicitly entangles manipulation artifacts with semantic content, causing shortcut learning and poor cross-domain generalization.

Approach

Model artifacts explicitly as a latent variable by training a shared encoder with pairwise BCE on edit-related vs. marginally matched non-related image pairs (using absolute feature differences), then use the resulting encoder as initialization for standard IML training.

Key results

• PAL initialization raises All-Avg F1 of ConvNeXt-Small from 0.5661 (ImageNet baseline) to 0.6266 and Cross-Avg from 0.5320 to 0.5868.
• PAL improves relative F1 across 11 architectures, from +5.72% (PSCC) to +14.0% (SAM-Base) and +10.68% for ConvNeXt-Small.
• PAL boundary activation increases ~5-7x over ImageNet initialization across six datasets (e.g., CASIAv1 0.48 to 3.03).

Surprising findings

• Absolute binary supervision on EditGroup-45K (Baseline-ABS) gives the best In-Avg but lowers Cross-Avg below the ImageNet baseline, indicating overfitting.
• Removing Self-Augmented negatives raises In-Domain F1 to 0.8521 while Cross-Domain drops, suggesting source-domain shortcuts.

Limitations

• Gains are measured by F1 under one protocol (Protocol-CAT) and the artifact-disentanglement claims rely on heuristic residual subtraction with an ImageNet encoder.
• Theoretical guarantees assume marginal matching, which is only approximately satisfied by the constructed negatives.

Experimental setup

Datasets: EditGroup-45K M^3 CASIA v2 IMD2020 FantasticReality Tampered COCO CASIA v1 COVERAGE NIST16 Columbia AutoSplice CocoGlide

Benchmarks: CASIA v1 COVERAGE Columbia NIST16 AutoSplice CocoGlide

Metrics: pixel-level F1 AUC

Baselines: Baseline (ImageNet init) Baseline-ABS CAT-Net PSCC MVSS Mesorch TruFor

Builds on: ForensicHub M^3

Closest papers by meaning

• Training-Free Image Manipulation Localization Using Diffusion Models (AAAI 2025 · similarity 0.87)
• Towards Modern Image Manipulation Localization: A Large-Scale Dataset and Novel Methods (CVPR 2024 · similarity 0.87)
• Can We Get Rid of Handcrafted Feature Extractors? SparseViT: Nonsemantics-Centered, Parameter-Efficient Image Manipulation Localization Through Spare-Coding Transformer (AAAI 2025 · similarity 0.86)
• Mesoscopic Insights: Orchestrating Multi-Scale & Hybrid Architecture for Image Manipulation Localization (AAAI 2025 · similarity 0.86)
• Less Supervision, Better Generalization: Weakly Supervised Fake Region Localization in Diffusion-Edited Images (NeurIPS 2026 · similarity 0.85)
• IDseq: Decoupled and Sequentially Detecting and Grounding Multi-Modal Media Manipulation (AAAI 2025 · similarity 0.85)