Research map / Label-efficient and robust learning / Recognition and long-tail
A Unifying Information-theoretic Perspective on Evaluating Generative Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Unifies kNN-based precision/recall metrics (PRC, Density & Coverage) under an information-theoretic lens and proposes a three-part metric (PCE, RCE, RE) separating fidelity, inter-class and intra-class diversity for generative image evaluation.
Problem
Existing precision and recall metrics for generative models are hard to compare and do not distinguish inter-class (mode dropping) from intra-class (mode shrinkage) diversity loss, nor fidelity failures (mode invention).
Approach
Reinterprets kNN-based precision/recall metrics as nonparametric divergence estimators (via kNN density-ratio estimation), and builds a metric from kNN entropy and cross-entropy estimators: precision cross-entropy (PCE), recall cross-entropy (RCE), and recall entropy (RE), which are additive per sample.
Key results
- PCE, PC and FD all show a positive correlation of at least 0.7 with human error rate on both CIFAR-10 and ImageNet, while Density shows a heavy negative correlation on CIFAR-10.
- Only RE is significantly sensitive to mode shrinkage among recall measures when varying CFG on DiT-XL-2.
- Density starts near 0.69 for a 2D Gaussian test and approaches 1.0 with increasing sample size, while PC stays essentially constant.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase generated variance with classifier-free guidance | DiT-XL-2 on ImageNet; metric behavior · ImageNet mode shrinkage experiment | Recall Entropy (RE) sensitivity to intra-class diversity | Helped |
| Increase classifier-free guidance | DiT-XL-2 on ImageNet; Precision Coverage (PC) · ImageNet mode shrinkage experiment | PC | Hurt |
| Increase classifier-free guidance | DiT-XL-2 on ImageNet; Density and PCE · ImageNet mode shrinkage experiment | fidelity response to mode shrinkage | Helped |
| Drop ImageNet classes from the generated dataset | ADMG-ADMU generated dataset; RCE, Coverage, and RC · ImageNet mode dropping experiment | sensitivity to mode dropping | Helped |
| Drop ImageNet classes from the generated dataset | ADMG-ADMU generated dataset; D, PC, PCE, and RE · ImageNet mode dropping experiment | stability under mode dropping | Helped |
| Evaluate realism metrics against human error rankings | CIFAR-10 and ImageNet; PCE, PC, and FD · CIFAR-10 and ImageNet | Pearson correlation with human error rate: ≥ 0.7; 0.08 | Helped |
| Use Density as the realism measure | CIFAR-10 and ImageNet · CIFAR-10 and ImageNet human-ranking experiment | correlation with human error rate | Mixed |
| Increase sample size | Density on shifted Gaussian distributions · Additional sample-size experiment | Density: approximately 0.69; y=1.0 | Hurt |
| Increase sample size | PC on shifted Gaussian distributions · Additional sample-size experiment | PC: 0.015 | No clear effect |
| Use Fréchet distance as a one-dimensional metric | Generative-model failure-mode evaluation · Mode shrinkage and mode dropping | ability to distinguish failure modes | Hurt |
Surprising findings
- Density shows a heavy negative correlation with human error rate on CIFAR-10 despite positive correlation on ImageNet.
- Precision Coverage decreases as generated samples become more clustered (higher CFG), contrary to expectation.
Limitations
- Requires a semantically meaningful embedding space; results depend on encoder quality.
- Bias bounds may be poor for extreme distributions with large K_max.
Experimental setup
Datasets: ImageNet CIFAR-10
Benchmarks: ImageNet CIFAR-10
Metrics: Pearson correlation with human error rate precision recall cross-entropy
Baselines: Density & Coverage Precision Recall Cover (PC, RC) Fréchet distance (FD) Improved Precision and Recall Rarity Score
Builds on: Precision Recall Cover Density and Coverage Improved Precision and Recall Precision-Recall for Distributions kNN divergence estimator (Wang et al.)
Closest papers by meaning
- Position: All Current Generative Fidelity and Diversity Metrics are Flawed (ICML 2025 · similarity 0.89)
- Efficient Precision and Recall Metrics for Assessing Generative Models using Hubness-aware Sampling (ICML 2024 · similarity 0.89)
- Towards a Scalable Reference-Free Evaluation of Generative Models (NeurIPS 2024 · similarity 0.88)
- An Interpretable Evaluation of Entropy-based Novelty of Generative Models (ICML 2024 · similarity 0.87)
- Generative Cross-Entropy: A Strictly Proper Loss for Data-Efficient Classification (NeurIPS 2026 · similarity 0.86)
- PQMass: Probabilistic Assessment of the Quality of Generative Models using Probability Mass Estimation (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.