Research map / Probabilistic and scientific modeling / Molecules and proteins
Position: All Current Generative Fidelity and Diversity Metrics are Flawed
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
The paper proposes six desiderata and a suite of sanity checks for generative fidelity and diversity metrics, and finds that all 11 evaluated single-number metrics fail many checks, arguing all current metrics are flawed.
Problem
Existing fidelity and diversity metrics for generative models (e.g., precision/recall variants) have known failure cases such as outlier sensitivity and unclear bounds, and it is unclear whether fixed metrics avoid other failures.
Approach
Distill known and new failure modes into simple sanity checks with programmatic passing criteria, linked to six desiderata, and evaluate many fidelity and diversity metrics together to show that all of them fail a substantial number of checks.
Key results
- Every one of the 11 evaluated fidelity and diversity metrics fails multiple sanity checks, supporting the position that all current metrics are flawed.
- Integrated alpha-precision reaches only about 0.5 in the 64-dimensional Gaussian mean difference check, so values near 0.5 can indicate good or bad data depending on dimension, showing unclear bounds.
- All metrics fail the hypercube varying sample size check (D3), so none converge with dataset size by 1000 points.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use current fidelity and diversity metrics to evaluate a difference in one dimension among many identical dimensions | Current generative fidelity and diversity metrics · One Disjoint Dim. + Many Identical Dim. | sanity-check pass | Hurt |
| Use current fidelity and diversity metrics to distinguish continuous numerical data from rounded discrete numerical data | Current generative fidelity and diversity metrics · Discrete Num. vs. Continuous Num. | sanity-check pass | Hurt |
| Vary the real and synthetic dataset sizes on overlapping uniform hypercubes | Current generative fidelity and diversity metrics · Hypercube, Varying Sample Size | sanity-check pass | Hurt |
| Scale one dimension in the tabular data check | I-Prec · Scaling One Dimension | sanity-check pass | Hurt |
| Scale one dimension in the tabular data check | Density, IAP, C-Prec, symPrec, P-Prec, I-Rec, Coverage, IBR, C-Rec, symRec, and P-Rec · Scaling One Dimension | sanity-check pass | Helped |
| Use metrics to detect outliers | I-Prec, symPrec, and I-Rec · Gaussian Mean Difference + Outlier | sanity-check pass | Hurt |
| Use metrics to detect mode collapse, dropping, or invention | I-Prec, P-Prec, and P-Rec · Mode Collapse; Mode Dropping + Invention; Sequential / Simultaneous Mode Dropping | sanity-check pass | Helped |
| Use symPrec and symRec to distinguish distributions on close hypersphere surfaces | symPrec and symRec · Hypersphere Surface | sanity-check pass | Helped |
| Use IAP and IBR to distinguish distributions on close hypersphere surfaces | IAP and IBR · Hypersphere Surface | sanity-check pass | Mixed |
| Use IBR to distinguish distributions on complex surrounding shapes | IBR · Sphere vs. Torus | sanity-check pass | Helped |
| Use C-Prec to distinguish distributions on complex surrounding shapes | C-Prec · Sphere vs. Torus | sanity-check pass | Hurt |
| Evaluate metrics with different real dataset sizes | Current generative fidelity and diversity metrics · Hypercube, Varying Sample Size | sanity-check pass | Hurt |
Surprising findings
- Density reaches values around 200 in high dimensions even for identical distributions, so high values do not indicate better synthetic data.
- All metrics fail the one-disjoint-dimension check, where a large difference in a single dimension among many identical dimensions goes undetected.
Limitations
- Sanity checks use simple artificial distributions rather than real data.
- Some checks may be irrelevant or have design flaws, as the authors acknowledge.
- Only single-number metrics were included; curve-valued metrics and topological metrics were excluded.
Experimental setup
Benchmarks: sanity checks (Gaussian mean/std difference, mode dropping/invention, hypercube sizes, hypersphere surface, sphere vs to
Metrics: pass/fail per sanity check against desiderata D1b, D2, D3, D
Baselines: I-Prec I-Rec Density Coverage IAP IBR C-Prec C-Rec symPrec symRec P-Prec P-Rec
Builds on: improved precision/recall density/coverage alpha-precision/beta-recall precision cover/recall cover symmetric precision/recall probabilistic precision/recall
Closest papers by meaning
- A Unifying Information-theoretic Perspective on Evaluating Generative Models (AAAI 2025 · similarity 0.89)
- Debiasing Synthetic Data Generated by Deep Generative Models (NeurIPS 2024 · similarity 0.88)
- Collapse or Thrive: Perils and Promises of Synthetic Data in a Self-Generating World (ICML 2025 · similarity 0.87)
- On the Stability of Iterative Retraining of Generative Models on their own Data (ICLR 2024 · similarity 0.86)
- Towards a Theoretical Understanding of Synthetic Data in LLM Post-Training: A Reverse-Bottleneck Perspective (ICLR 2025 · similarity 0.85)
- An Interpretable Evaluation of Entropy-based Novelty of Generative Models (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.