Research map / Image generation / Autoregressive and latent generation
Anomaly Score: Evaluating Generative Models and Individual Generated Images based on Complexity and Vulnerability
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Anomaly Score (AS), a 2D KS-statistic over complexity and vulnerability of feature-space neighborhoods, to evaluate generative models, and AS-i, a per-image ratio score, to assess individual generated images.
Problem
Feature-distance metrics such as FID often misalign with human judgments of naturalness of generated images, and few metrics evaluate individual generated images.
Approach
Characterize the representation space around an image by how non-linear it is under noise trajectories (complexity) and how easily features change under adversarial input perturbation (vulnerability); compare real vs generated distributions of these with a 2D KS statistic, and score single images by vulnerability divided by complexity.
Key results
- AS with DINO-V2 reaches -0.81 PCC with human error rate on the overall dataset, versus -0.54 for FID (Table 3 reports -0.806 vs -0.540).
- AS-i correlates with human natural-judgment proportion at -0.88 PCC, while rarity score (-0.49) and realism score (0.63) correlate less well.
- Vulnerability correlates strongly with MVT recognition levels across ViT-S, ConvNeXt-tiny, and DINO-V2 feature models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use complexity and vulnerability together instead of complexity alone (ablation) | Anomaly Score · Overall generated datasets | PCC with human error rate: -0.376 -> -0.806 | Helped |
| Use complexity and vulnerability together instead of vulnerability alone (ablation) | Anomaly Score · Overall generated datasets | PCC with human error rate: -0.790 -> -0.806 | Helped |
| Use Anomaly Score instead of FID | Generative model evaluation with DINO-V2 · Overall generated datasets | PCC with human error rate: -0.540 -> -0.806 | Helped |
| Increase the AS-i level | InsGen images · Subjective naturalness test | Proportion judged natural: 0.75 -> 0.25 | Hurt |
Surprising findings
- Complexity alone with ConvNeXt-tiny and DINO-V2 gives near-zero or positive correlation with human error rate on FFHQ (0.11 and -0.07 PCC), so the 2D combination is needed.
- The realism score assigns the highest realism to an unnatural image in the leftmost example of Figure 1, illustrating misalignment with human judgment.
Limitations
- Requires about 20 forward passes per image (K=10 noise steps plus J=10 attack steps), versus one for FID.
- Correlation with human judgment depends on the choice of feature model; ConvNeXt-tiny correlates only -0.30 on CIFAR10-level analysis and ViT-S on FFHQ, so alignment is inconsistent across feature models.
- Subjective test for AS-i uses only 14 participants and 20 images per level, a small-scale study.
Experimental setup
Datasets: CIFAR10 ImageNet FFHQ MVT dataset dgm-eval generated datasets
Benchmarks: dgm-eval human error rate MVT dataset
Metrics: Pearson correlation coefficient Spearman rank correlation coefficient human error rate human natural proportion
Baselines: FID realism score rarity score 1D-Complexity 1D-Vulnerability
Builds on: FID PGD attack realism score rarity score precision and recall
Closest papers by meaning
- Rarity Score : A New Metric to Evaluate the Uncommonness of Synthesized Images (ICLR 2023 · similarity 0.89)
- Image Generation Diversity Issues and How to Tame Them (CVPR 2025 · similarity 0.87)
- Beyond Generation: A Diffusion-based Low-level Feature Extractor for Detecting AI-generated Images (CVPR 2025 · similarity 0.86)
- Detecting Generated Images by Fitting Natural Image Distributions (NeurIPS 2025 · similarity 0.86)
- Manifold Induced Biases for Zero-shot and Few-shot Detection of Generated Images (ICLR 2025 · similarity 0.85)
- GenAI Arena: An Open Evaluation Platform for Generative Models (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.