Research map / Probabilistic and scientific modeling / Variational and generative sampling
An Interpretable Evaluation of Entropy-based Novelty of Generative Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes the Kernel-based Entropic Novelty (KEN) score, computed from the spectrum of a differential kernel covariance matrix, to quantify mode-based novelty of a generative model relative to a reference dataset and to identify the novel modes.
Problem
Evaluating how much a generative model's samples express sample types (modes) more frequently than a reference distribution; existing metrics (FID, precision/recall, rarity) measure fidelity, diversity, or single-sample uncommonness rather than distribution-level novelty.
Approach
Take the eta-differential kernel covariance matrix C_X - eta*C_Y; its positive eigenvalues approximate the frequencies of modes over-represented in the test model, and their entropy (scaled by total mass) gives the KEN score. A Cholesky-based kernel-matrix reduction computes it from pairwise kernel similarities only.
Key results
- On 2D Gaussian mixtures, adding two novel modes raises the KEN score from 0.74 to 1.40, while the diversity-based baselines (improved Recall, Coverage, RKE) decrease as the novel-mode ratio alpha grows.
- Among FFHQ-trained generators, InsGen has the highest average pairwise KEN (1.37), and VDVAE the lowest (0.94).
- KEN and reverse KEN are asymmetric: adding four extra reference modes left KEN nearly unchanged (1.40 vs 1.39) while R-KEN rose from 1.40 to 1.83.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add two novel modes to the test distribution | KEN evaluation of a Gaussian mixture · Synthetic Gaussian mixtures | KEN: 0.74 to 1.40 | Helped |
| transfer weight from novel modes to common modes | KEN evaluation of a Gaussian mixture · Synthetic Gaussian mixtures | KEN: 1.40 to 0.92 | Hurt |
| increase common-mode frequencies from 0.25 to 0.4 | KEN evaluation of a Gaussian mixture · Synthetic Gaussian mixtures | number of outstanding eigenvalues: 0.25 to 0.4 | Mixed |
| use identical test and reference distributions | KEN evaluation · Synthetic Gaussian mixtures | KEN: nearly 0 | Hurt |
| add four extra reference modes | KEN and Reverse KEN evaluation · Synthetic Gaussian mixtures | KEN: 1.40 vs. 1.39 | No clear effect |
| add four extra reference modes | Reverse KEN evaluation · Synthetic Gaussian mixtures | Reverse KEN: 1.40 to 1.83 | Helped |
| increase the proportion of novel aquatic modes | KEN, FLD, Rarity, and diversity-based metrics · ImageNet-1K terrestrial-animal reference and aquatic-life test distributions | KEN | Helped |
| increase the proportion of novel aquatic modes | diversity-based baseline metrics · ImageNet-1K terrestrial-animal reference and aquatic-life test distributions | diversity-based scores | Hurt |
| increase novel modes from 1 to 3 aquatic animals | KEN compared with sample-based FLD and Rarity · ImageNet-1K terrestrial-animal reference and aquatic-life test distributions | KEN | Helped |
| increase novel modes from 1 to 3 aquatic animals | FLD and Rarity · ImageNet-1K terrestrial-animal reference and aquatic-life test distributions | sample-based novelty scores | No clear effect |
| apply the truncation trick (ablation) | StyleGAN-XL · FFHQ | KEN | Hurt |
| use Inception-V3 rather than DINOv2 or CLIP embedding | KEN model ranking · FFHQ-trained generative models | average KEN ranking | Mixed |
Surprising findings
- Novelty and diversity metrics move in opposite directions: diversity-based baselines decrease as the test distribution becomes more novel.
- KEN is asymmetric between test and reference distributions, so reverse KEN behaves differently.
Limitations
- Computational cost is O((n+m)^3) from eigendecomposition, limiting scalability to large sample sizes.
- Results depend on the choice of embedding (e.g., Inception-V3 bias), and the paper does not analyze this in depth.
- Theory covers only mixtures with well-separable components and a Gaussian kernel with a chosen bandwidth.
Experimental setup
Datasets: CIFAR-10 ImageNet-1K CelebA FFHQ AFHQ Wildlife dataset
Benchmarks: synthetic 2D Gaussian mixtures ImageNet-1K subsets FFHQ-trained generative models
Metrics: KEN score Reverse KEN IS FID Precision Recall Density Coverage RKE FLD
Baselines: Improved Recall Coverage RKE FLD Rarity score FID Precision Density
Builds on: Fréchet Inception Distance Density and Coverage Rarity score
Closest papers by meaning
- Towards a Scalable Reference-Free Evaluation of Generative Models (NeurIPS 2024 · similarity 0.88)
- Unveiling Differences in Generative Models: A Scalable Differential Clustering Approach (CVPR 2025 · similarity 0.88)
- A Unifying Information-theoretic Perspective on Evaluating Generative Models (AAAI 2025 · similarity 0.87)
- Image Generation Diversity Issues and How to Tame Them (CVPR 2025 · similarity 0.86)
- Position: All Current Generative Fidelity and Diversity Metrics are Flawed (ICML 2025 · similarity 0.85)
- Towards Theoretical Understandings of Self-Consuming Generative Models (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.