Research map / Learning theory and algorithms / Kernels and regression theory
Towards a Scalable Reference-Free Evaluation of Generative Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FKEA, a random-Fourier-feature approximation of the VENDI and RKE reference-free diversity entropy scores that scales linearly in sample size, with eigenvector-based mode identification.
Problem
Reference-free diversity evaluation of generative models via VENDI and RKE entropy scores requires eigendecomposition of an n x n kernel matrix, which is computationally prohibitive for large sample sizes.
Approach
Approximate the kernel covariance matrix with random Fourier features of a shift-invariant (Gaussian) kernel, so that the eigenspectrum and entropy scores are computed from a 2r x 2r matrix built in O(n) time, with proven eigenvalue/eigenvector error bounds and proxy eigenvectors that reveal sample modes.
Key results
- FKEA scales to n=250k ImageNet samples (FKEA-RKE 238 s at 2r=8000) whereas exact VENDI and RKE run out of memory beyond 30k samples.
- Theory: FKEA needs RFF size 2r = O(log n / epsilon^2) for eigenvalue and eigenvector approximation with high probability.
- On ImageNet generative models, FKEA scores rank DiT-XL-2 highest in RKE (855.8) and VENDI-1 (8626.5), and RQ-Transformer and BigGAN lowest.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FKEA-RKE instead of direct RKE computation | RKE on ImageNet · ImageNet, n=10k | time (sec): 217 -> 7 | Helped |
| Use FKEA-VENDI instead of direct VENDI computation | VENDI on ImageNet · ImageNet, n=10k | time (sec): 286 -> 11 | Helped |
| Use FKEA to approximate entropy scores | RKE and VENDI on ImageNet · ImageNet, n=250k | computability at sample size | Helped |
| Increase the number of datapoints from 10k to larger sample sizes | FKEA clustering · ImageNet | cluster quality: 10k, 50k, 100k, and 250k | Helped |
| Include samples from more labels | FKEA diversity scores · ImageNet | diversity score | Helped |
| Include samples from more classes | FKEA diversity scores · UCF101 | diversity score | Helped |
Surprising findings
- Reference-based FID, Recall and Coverage fail to indicate higher diversity for colored elephant images that look visibly more diverse, while reference-free VENDI and RKE increase.
Limitations
- Analysis restricted to shift-invariant kernels, so not applicable to polynomial kernels and other non-shift-invariant kernels.
- Results depend on the quality of the underlying embedding space (DINOv2, I3D, text-embedding-3-large).
- Text clustering is harder to interpret, and the bandwidth must be tuned per dataset.
Experimental setup
Datasets: ImageNet MNIST colored MNIST FFHQ AFHQ MS-COCO Fashion-MNIST UCF101 Kinetics-400 Wikipedia CNN/Dailymail CMU Movie Corpus
Benchmarks: ImageNet MNIST FFHQ AFHQ MS-COCO Fashion-MNIST UCF101 Wikipedia CNN/Dailymail CMU Movie Corpus
Metrics: VENDI RKE FKEA-VENDI FKEA-RKE runtime IS FID precision recall density
Baselines: VENDI RKE FID Inception Score Precision/Recall Density/Coverage
Builds on: VENDI RKE random Fourier features kernel PCA
Closest papers by meaning
- An Interpretable Evaluation of Entropy-based Novelty of Generative Models (ICML 2024 · similarity 0.88)
- A Unifying Information-theoretic Perspective on Evaluating Generative Models (AAAI 2025 · similarity 0.88)
- Unveiling Differences in Generative Models: A Scalable Differential Clustering Approach (CVPR 2025 · similarity 0.87)
- Image Generation Diversity Issues and How to Tame Them (CVPR 2025 · similarity 0.86)
- A Bias-Variance-Covariance Decomposition of Kernel Scores for Generative Models (ICML 2024 · similarity 0.86)
- PQMass: Probabilistic Assessment of the Quality of Generative Models using Probability Mass Estimation (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.