Research map / Image generation / Autoregressive and latent generation
Semi-Truths: A Large-Scale Dataset of AI-Augmented Images for Evaluating Robustness of AI-Generated Image detectors
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Semi-Truths, a large dataset of ~1.47M AI-augmented images with masks and metadata, plus a plug-and-play augmentation and evaluation pipeline, and uses it to stress-test six AI-generated image detectors.
Problem
Existing AI-generated image detectors are mostly trained and evaluated on fully synthetic, often face-centric images, so their robustness to targeted, localized, magnitude-controlled AI edits and to data distribution shifts is unknown.
Approach
Create a large set of real images paired with targeted diffusion-based edits (conditional inpainting and prompt-based editing) whose magnitude of change is quantified by area-ratio, SSIM/MSE-based, and semantic metrics, with rich metadata enabling per-factor stress tests of detectors.
Key results
- Of six detectors, DINOv2, CNNSpot and DIRE perform near chance (fake recall around 0.00-0.01%), while CrossEfficientViT, UniversalFakeDetect and DE-FAKE are selected for further analysis.
- UniversalFakeDetect's performance drops with increasing scene diversity and complexity, while DE-FAKE stays fairly robust across scene variations.
- Human study (800 image pairs, 145 annotators) found Area Ratio correlates most with human perception (Spearman 0.50), while LPIPS and SSIM correlate weakly (0.19 and -0.17).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Evaluate across source data distributions | CrossEfficientViT · Semi-Truths source benchmarks | Fake-class recall | Mixed |
| Evaluate across source data distributions | DE-FAKE · Semi-Truths source benchmarks | Fake-class recall | Mixed |
| Evaluate across source data distributions | UniversalFakeDetect · SUN RGBD and CityScapes | Fake-class recall | Hurt |
| Increase scene diversity and complexity | UniversalFakeDetect · Semi-Truths scenes | Fake-class recall | Hurt |
| Increase scene diversity and complexity | CrossEfficientViT · Semi-Truths scenes | Fake-class recall | Helped |
| Vary scene diversity and complexity | DE-FAKE · Semi-Truths scenes | Fake-class recall | No clear effect |
| Use Kandinsky 2.2 rather than StableDiffusion v1.5 for augmentation | UniversalFakeDetect · Semi-Truths diffusion-model variants | Fake-class recall: 10% | Helped |
| Use Kandinsky 2.2 rather than StableDiffusion v1.4 for augmentation | CrossEfficientViT · Semi-Truths diffusion-model variants | Fake-class recall: 12% | Hurt |
| Use prompt-based editing rather than inpainting | UniversalFakeDetect · Semi-Truths augmentation techniques | Fake-class recall | Hurt |
| Use inpainting rather than prompt-based editing | CrossEfficientViT and DE-FAKE · Semi-Truths augmentation techniques | Fake-class recall | Hurt |
| Increase Area Ratio | CrossEfficientViT · Semi-Truths augmentation magnitudes | Fake-class recall | Hurt |
| Increase Area Ratio | UniversalFakeDetect · Semi-Truths augmentation magnitudes | Fake-class recall | Helped |
Surprising findings
- Detectors trained on general or face data show large sensitivity shifts across data distributions, e.g. CrossEfficientViT (trained on GAN faces) fails on non-face benchmarks, though it pre-filters non-face images.
- Standard metrics like LPIPS and SSIM show little correlation with human-perceived magnitude of change, while area ratio does.
Limitations
- Inpainting relies on manually provided segmentation masks from existing benchmarks, limiting usability.
- Zero-shot LLaMA-7B and LLaVA-Mistral-7B perturbations produced many poor-quality outputs requiring filtering.
- Semantic change metrics (LPIPS, Sentence Similarity, DreamSim) may carry model biases; some pixel-level changes are imperceptible to humans.
Experimental setup
Datasets: CityScapes SUN RGBD CelebA-HQ Human Parsing ADE20K OpenImages
Benchmarks: Semi-Truths evaluation set
Metrics: precision recall F1 AUC-ROC AP Pearson correlation Kendall-Tau Spearman correlation
Baselines: DINOv2 CNNSpot DIRE CrossEfficientViT UniversalFakeDetect DE-FAKE
Builds on: LANCE InstructPix2Pix DE-FAKE UniversalFakeDetect DIRE
Closest papers by meaning
- BIAS-ID: A Framework for Analyzing Transformation Biases in AI-Generated Image Detectors (NeurIPS 2026 · similarity 0.87)
- Are High-Quality AI-Generated Images More Difficult for Models to Detect? (ICML 2025 · similarity 0.86)
- Community Forensics: Using Thousands of Generators to Train Fake Image Detectors (CVPR 2025 · similarity 0.86)
- Is Artificial Intelligence Generated Image Detection a Solved Problem? (NeurIPS 2025 · similarity 0.86)
- A Bias-Free Training Paradigm for More General AI-generated Image Detection (CVPR 2025 · similarity 0.85)
- SALART-VQA: Diagnosing Whether VLMs Understand Salient Artifacts in Generated Images (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.