Research map / Architectures and efficiency / RNNs and normalization
Efficient Precision and Recall Metrics for Assessing Generative Models using Hubness-aware Sampling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes efficient precision and recall (eP&R) metrics for generative model evaluation that use hubness-aware sampling and approximate k-NN to approximate the original P&R at much lower time and memory cost.
Problem
Original precision and recall metrics for generative models require O(n^2 log n) time and O(n^2) space due to pairwise k-NN distances, becoming infeasible on commodity hardware for large datasets.
Approach
Identifies two redundancies in P&R (ratio computation and inside/outside manifold identification) and shows that a small set of high-hubness samples can replace the full sets, with approximate k-NN (IVF-PQ) used for hub identification since hub selection is insensitive to exact k-NN results.
Key results
- eP&R matches original P&R on StyleGAN2 FFHQ with relative errors around 1% (e.g., precision 0.719 vs 0.716, recall 0.501 vs 0.493).
- On FFHQ, eP&R total time drops from 399s to 165s serial (144s to 75s parallel) and peak memory from 19.90 GB to 14.24 GB.
- Ablation shows both redundancy-removal components and approximate k-NN contribute; eP&R reduces time from 144s to 75s (parallel) on FFHQ.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use hubness-aware sampling and approximate k-NN for eP&R instead of full-sample original P&R | Precision and recall evaluation · FFHQ, LSUN-Car, LSUN-Church, LSUN-Cat, LSUN-Horse; StyleGAN3, Projected-GAN, VQ-VAE-2, Latent Diffusion on FFHQ | precision and recall: Error(%): 0.4%–2.3% | No clear effect |
| Use eP&R instead of original P&R | Precision and recall evaluation on FFHQ (70k images) · FFHQ | parallel runtime: 144s -> 75s | Helped |
| Use eP&R instead of original P&R | Precision and recall evaluation on FFHQ (70k images) · FFHQ | peak memory: 19.90 GB -> 14.24 GB | Helped |
| Remove representative-sample ratio estimation (Observation 4.1) (ablation) | eP&R · FFHQ | precision: 0.719±0.002 -> 0.715±0.005 | No clear effect |
| Remove representative-sample ratio estimation (Observation 4.1) (ablation) | eP&R · FFHQ | recall: 0.501±0.001 -> 0.497±0.004 | No clear effect |
| Remove manifold inside/outside redundancy reduction (Observation 4.2) (ablation) | eP&R · FFHQ | precision: 0.719±0.002 -> 0.708±0.005 | No clear effect |
| Remove manifold inside/outside redundancy reduction (Observation 4.2) (ablation) | eP&R · FFHQ | recall: 0.501±0.001 -> 0.501±0.005 | No clear effect |
| Remove approximate k-NN for hub identification (ablation) | eP&R with Observations 4.1 and 4.2 · FFHQ | precision: 0.719±0.002 -> 0.719±0.002 | No clear effect |
| Remove approximate k-NN for hub identification (ablation) | eP&R with Observations 4.1 and 4.2 · FFHQ | recall: 0.501±0.001 -> 0.494±0.001 | No clear effect |
| Use k=5 instead of k=3 (ablation) | eP&R · FFHQ | precision: 0.719±0.001 -> 0.813±0.002 | No clear effect |
| Use k=5 instead of k=3 (ablation) | eP&R · FFHQ | recall: 0.494±0.001 -> 0.615±0.002 | No clear effect |
| Increase hub threshold from t=3 to t=5 (ablation) | eP&R · FFHQ | precision: 0.719±0.002 -> 0.730±0.001 | Mixed |
Surprising findings
- Using only hubness-aware sampling for the ratio or manifold steps alone does not save time or memory; savings come from combining them with approximate k-NN.
Limitations
- Distance computation between hubs and full sets (Stage 3) remains a major time cost, as acknowledged by authors.
- Accuracy trades off with hubness threshold t: higher t is faster but increases error (e.g., t=5 gives precision 0.730 vs 0.716 baseline).
- Evaluated only on StyleGAN2-family and a few image generators at 70k-1.5M scale on image datasets.
Experimental setup
Datasets: FFHQ LSUN-Car LSUN-Church LSUN-Cat LSUN-Horse
Benchmarks: FFHQ LSUN-Car LSUN-Church LSUN-Cat LSUN-Horse
Metrics: precision recall relative approximation error time (s) memory (GB)
Baselines: original precision and recall (Kynkäänniemi et al., 2019) Sajjadi et al. 2018 P&R Simon et al. 2019 P&R
Builds on: Kynkäänniemi et al. 2019 improved precision and recall hubness phenomenon IVF-PQ
Closest papers by meaning
- A Unifying Information-theoretic Perspective on Evaluating Generative Models (AAAI 2025 · similarity 0.89)
- Sampling is as easy as keeping the consistency: convergence guarantee for Consistency Models (ICML 2024 · similarity 0.86)
- Towards a Scalable Reference-Free Evaluation of Generative Models (NeurIPS 2024 · similarity 0.86)
- Distance Marching for Generative Modeling (NeurIPS 2026 · similarity 0.86)
- A Geometric Framework for Understanding Memorization in Generative Models (ICLR 2025 · similarity 0.85)
- Improved Training Technique for Latent Consistency Models (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.