Research map / Image generation / Autoregressive and latent generation
A Lottery Ticket Hypothesis Approach with Sparse Fine-tuning and MAE for Image Forgery Detection and Localization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Forgery MAE (FMAE), which uses lottery-ticket-style selection of forgery-sensitive MAE parameters for sparse fine-tuning plus a mixture-of-experts noise extractor (SRM, Bayar, NoisePrint++) for image forgery detection and localization.
Problem
Image forgery detection and localization (IFDL) generalizes poorly to out-of-distribution, unseen forgery types, since existing methods design customized networks for specific tampering artifacts.
Approach
Keep natural-image priors from a pretrained MAE by sparsely fine-tuning only the parameters that differ most between forged-image and authentic-image masked-image-modeling pretraining (lottery ticket selection, rewound to original weights), and inject multi-source noise features from a mixture-of-experts noise extractor into the layers holding those parameters.
Key results
- Average pixel-level F1 of 0.493 and AUC of 0.872 for localization across eight test sets, the best average F1 among compared methods (DiffForensics 0.466).
- Image-level detection average ACC of 0.806 vs DiffForensics 0.770 on Casiav1.0, Columbia, IMD2020, AutoSplice; best ACC on the average, with AUC 0.902 second-best behind DiffForensics 0.916.
- Ablation on IMD2020 and AutoSplice: full setup F1 0.491/0.639 vs random 70% unfreezing (w/o LTSFT) 0.375/0.427.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FMAE instead of DiffForensics | image forgery localization · Casiav1.0 | pixel-level F1: .517 -> .612 | Helped |
| Use FMAE instead of DiffForensics | image forgery localization · Columbia | pixel-level F1: .912 -> .881 | Hurt |
| Use FMAE instead of DiffForensics | image forgery localization · AutoSplice | pixel-level F1: .507 -> .639 | Helped |
| Use FMAE instead of DiffForensics | image forgery localization · Average across test datasets | pixel-level F1: .466 -> .493 | Helped |
| Use FMAE instead of DiffForensics | image forgery detection · Average across test datasets | image-level ACC: .770 -> .806 | Helped |
| Use FMAE instead of DiffForensics | image forgery detection · Average across test datasets | image-level AUC: .916 -> .902 | Hurt |
| Use FMAE instead of DiffForensics | image forgery detection · Average across test datasets | image-level EER: .131 -> .149 | Hurt |
| Replace winning-ticket parameter selection with randomly unfreezing 70% of parameters (ablation) | FMAE · IMD2020 | pixel-level F1: .375 -> .491 | Hurt |
| Replace winning-ticket parameter selection with randomly unfreezing 70% of parameters (ablation) | FMAE · AutoSplice | pixel-level F1: .427 -> .639 | Hurt |
| Add the MoENE multi-source noise extractor (ablation) | FMAE · IMD2020 | pixel-level F1: .429 -> .491 | Helped |
| Add the MoENE multi-source noise extractor (ablation) | FMAE · AutoSplice | pixel-level F1: .560 -> .639 | Helped |
| Feed forgery information into the layers containing winning-ticket parameters (ablation) | FMAE · IMD2020 | pixel-level F1: .445 -> .491 | Helped |
Surprising findings
- Unfreezing all MAE encoder parameters loses the natural-image prior and generalizes worse than sparse fine-tuning, with 70% unfrozen being the best balance point.
- Ours does not beat all baselines on every dataset; e.g., it is not best on Columbia localization F1 and detection AUC on some sets.
Limitations
- Forgery-sensitive parameter selection requires two full MAE pretraining runs on forged and authentic data before fine-tuning.
- Training uses only Casiav2.0 and FantasticReality; generalization evaluated on a limited set of test datasets.
- Detection results are mixed: best accuracy but second-best AUC and EER on the average.
Experimental setup
Datasets: CASIA v2.0 FantasticReality CASIA v1.0 Columbia NIST16 IMD2020 DSO-1 Korus AutoSplice OpenForensics
Benchmarks: CASIA v1.0 Columbia NIST16 IMD2020 DSO-1 Korus AutoSplice OpenForensics
Metrics: pixel-level F1 pixel-level AUC image-level ACC image-level AUC image-level EER
Baselines: ManTraNet SPAN MVSS-Net CAT-Net PSCC-Net HiFi-Net DiffForensics
Builds on: Masked Autoencoder Lottery Ticket Hypothesis ComposableSFT MVSS-Net DiffForensics
Closest papers by meaning
- Learning Discriminative Noise Guidance for Image Forgery Detection and Localization (AAAI 2024 · similarity 0.89)
- FakeShield: Explainable Image Forgery Detection and Localization via Multi-modal Large Language Models (ICLR 2025 · similarity 0.87)
- Identity-Aware Vision-Language Model for Explainable Face Forgery Detection (AAAI 2026 · similarity 0.87)
- SUMI-IFL: An Information-Theoretic Framework for Image Forgery Localization with Sufficiency and Minimality Constraints (AAAI 2025 · similarity 0.87)
- DiffForensics: Leveraging Diffusion Prior to Image Forgery Detection and Localization (CVPR 2024 · similarity 0.86)
- Reinforced Multi-teacher Knowledge Distillation for Efficient General Image Forgery Detection and Localization (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.