Research map / Image generation / Autoregressive and latent generation
Forgery-aware Adaptive Transformer for Generalizable Synthetic Image Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FatFormer, which adapts a frozen-then-tuned CLIP ViT with a forgery-aware adapter (image + DWT frequency attention) and language-guided contrastive alignment to detect GAN and diffusion-generated images from ProGAN-only training.
Problem
Generalizable synthetic image detection: detectors trained on one generator (ProGAN) fail on unseen GANs and diffusion models, and fixed frozen pre-trained paradigms (e.g., UniFD) learn insufficient forgery representations.
Approach
Instead of freezing CLIP and training a linear probe, insert forgery-aware adapters (image convolution branch plus DWT-based inter/intra-band attention) into the ViT stages and supervise them with language-guided contrastive alignment between patch tokens, text prompts and image CLS tokens.
Key results
- Trained on 4-class ProGAN, FatFormer reaches 98.4% ACC / 99.7% AP on the GANs dataset, versus 89.1% / 98.3% for UniFD.
- On the diffusion dataset, FatFormer achieves 95.0% ACC / 98.8% AP, surpassing UniFD (85.4% / 94.6%) by 9.6% ACC.
- Removing the forgery-aware adapter drops ACC_M to 91.5 (LGA only) and removing LGA drops to 95.3, with zero-shot CLIP at 66.6%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Adapt CLIP with FatFormer instead of using the fixed UniFD paradigm | CLIP ViT-L/14 · GANs dataset, 4-class ProGAN; mean | ACCM: 89.1% -> 98.4%; 9.3% ACC | Helped |
| Adapt CLIP with FatFormer instead of using the fixed UniFD paradigm | CLIP ViT-L/14 · Diffusion models; mean | ACC: 85.4% -> 95.0%; 9.6% ACC | Helped |
| Add frequency-domain features to the forgery-aware adapter (ablation) | FatFormer · GANs dataset, 4-class ProGAN; mean | ACCM: 95.4 -> 98.4; over −3.0% ACC gaps | Helped |
| Add image-domain features to the forgery-aware adapter (ablation) | FatFormer · GANs dataset, 4-class ProGAN; mean | ACCM: 97.3 -> 98.4 | Helped |
| Add inter-band attention alongside intra-band attention (ablation) | FatFormer frequency forgery extractor · GANs dataset, 4-class ProGAN; mean | ACCM: 97.4 -> 98.4 | Helped |
| Add intra-band attention alongside inter-band attention (ablation) | FatFormer frequency forgery extractor · GANs dataset, 4-class ProGAN; mean | ACCM: 96.6 -> 98.4 | Helped |
| Use image and text prompts with vanilla contrastive objectives instead of image-only linear probing (ablation) | FatFormer · GANs dataset, 4-class ProGAN; mean | ACCM: 95.3 -> 96.4 | Helped |
| Use augmented contrastive objectives instead of vanilla contrastive objectives (ablation) | FatFormer · GANs dataset, 4-class ProGAN; mean | ACCM: 2.0% ACC gain | Helped |
| Use auto context embeddings instead of a fixed hand-crafted template (ablation) | FatFormer text prompts · GANs dataset, 4-class ProGAN; mean | ACCM: 0.9% ACC | Helped |
| Condition auto context embeddings on patch tokens instead of the CLS token (ablation) | FatFormer text prompts · GANs dataset, 4-class ProGAN; mean | ACCM: 98.1 -> 98.4 | Helped |
| Add the forgery-aware adapter (ablation) | FatFormer · GANs dataset, 4-class ProGAN; mean | ACCM: −6.9% ACC | Helped |
| Add language-guided alignment instead of binary labels alone (ablation) | FatFormer · GANs dataset, 4-class ProGAN; mean | ACCM: 3.1% ACC and 0.5% AP gains | Helped |
Surprising findings
- A frozen CLIP with a linear probe (UniFD) yields only about 85% ACC on diffusion models, similar to PatchFor, despite strong pre-training.
- Zero-shot CLIP with text prompts reaches only 66.6% ACC mean, showing that the adaptation, not the prompt alone, drives performance.
Limitations
- Performance still lags on some diffusion models, e.g., Guided (76.1% ACC).
- Distinctions between diffusion-generated and GAN-generated image forgeries are not yet elucidated.
- Pretext tasks tailored for synthetic image detection in pre-training are not explored.
Experimental setup
Datasets: ProGAN 4-class training data (Wang et al.) GANs test set (Wang et al.) diffusion model dataset (Ojha et al. / Corvi et al.)
Benchmarks: GANs dataset (Wang et al.) diffusion model dataset (Ojha et al.)
Metrics: accuracy (ACC) average precision (AP) mean ACC (ACC_M) mean AP (AP_M)
Baselines: UniFD LGrad Wang et al. CNN detector F3Net FrePGAN BiHPF Frank Durall PatchFor Blend
Builds on: UniFD CLIP CoOp F3Net LGrad
Closest papers by meaning
- Identity-Aware Vision-Language Model for Explainable Face Forgery Detection (AAAI 2026 · similarity 0.88)
- Forensics Adapter: Adapting CLIP for Generalizable Face Forgery Detection (CVPR 2025 · similarity 0.88)
- FakeShield: Explainable Image Forgery Detection and Localization via Multi-modal Large Language Models (ICLR 2025 · similarity 0.86)
- Forensics-Bench: A Comprehensive Forgery Detection Benchmark Suite for Large Vision Language Models (CVPR 2025 · similarity 0.86)
- A Lottery Ticket Hypothesis Approach with Sparse Fine-tuning and MAE for Image Forgery Detection and Localization (AAAI 2025 · similarity 0.85)
- CO-SPY: Combining Semantic and Pixel Features to Detect Synthetic Images by AI (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.