Research map / Probabilistic and scientific modeling / Variational and generative sampling

PQMass: Probabilistic Assessment of the Quality of Generative Models using Probability Mass Estimation

ICLR 2025 · Probabilistic and scientific modeling › Variational and generative sampling · approach: Score-based generative modeling analysis · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes PQMass, a likelihood-free two-sample test that bins samples from two distributions into Voronoi cells and applies a Pearson chi-squared test to assess generative model quality (fidelity, diversity, novelty).

Problem

Evaluating generative models requires rigorous, scalable metrics for fidelity, diversity, and novelty; likelihood-based metrics are noisy and sample-based metrics like FID/FLD either miss novelty or need feature extraction to work in high dimensions.

Approach

Partition the sample space into Voronoi cells defined by random reference points drawn from both sample sets, count points per cell, and compare the two count vectors with a Pearson chi-squared test giving a p-value; no density assumptions, no auxiliary model training, and works directly in pixel space.

Key results

• On ImageNet with added Gaussian noise, PQMass chi-squared rises from ~96 (no noise) to ~4495 (variance 1.00), while FLD is nearly flat from 9.29 to ~42.5 and less sensitive in the low-noise regime.
• In the small-mode CIFAR-10 experiment, PQMass detects a truck mode at alpha=0.01 with a chi-squared value of 100.60 vs 100.41 baseline, whereas FLD stays near -4 throughout.
• On the 2D Gaussian mixture sampling benchmark, PQMass ranks MCMC (1621) worse than FAB (678) and GGNS (109), in agreement with RBF MMD and JSD.

Measured effects reported in the paper

Surprising findings

• FLD returns negative values on null tests (e.g., -21.72 for Probes ground truth vs ground truth), indicating instability for high-dimensional data.
• FID and FLD give contradictory out-of-distribution verdicts on JWST samples, while PQMass gives a consistent result.

Limitations

• Requires many i.i.d. samples; statistics are noisy with few samples.
• Results depend on the chosen distance metric, which is hard to define for modalities such as natural language.
• Sensitivity is limited in the extremes of number of regions (n_R=1 or n_R near number of samples); a fixed tessellation may fail to discriminate, so repeated random tessellations are needed.

Experimental setup

Datasets: MNIST CIFAR-10 ImageNet FFHQ256 Probes JWST darks SKIRT TNG Adult Census ESM Metagenomic Atlas

Benchmarks: CIFAR-10 human error rate MNIST mode dropping ImageNet Gaussian noise detection Neal's funnel 2D Gaussian mixture sampling Probes/JWST/SKIRT TNG astrophysics

Metrics: chi-squared PQMass statistic p-value FID FLD Wasserstein-2 RBF MMD Jensen-Shannon divergence authenticity score

Baselines: FID FLD Wasserstein distance (W2) RBF MMD Jensen-Shannon divergence authenticity score

Builds on: Tests of Accuracy with Random Points (TARP) Feature Likelihood Divergence Fréchet Inception Distance

Closest papers by meaning

• A Likelihood Based Approach to Distribution Regression Using Conditional Deep Generative Models (ICML 2025 · similarity 0.86)
• A Unifying Information-theoretic Perspective on Evaluating Generative Models (AAAI 2025 · similarity 0.86)
• Towards a Scalable Reference-Free Evaluation of Generative Models (NeurIPS 2024 · similarity 0.86)
• Statistically Optimal Generative Modeling with Maximum Deviation from the Empirical Distribution (ICML 2024 · similarity 0.85)
• Image Generation Diversity Issues and How to Tame Them (CVPR 2025 · similarity 0.85)
• Conformal Prediction Beyond the Seen: A Missing Mass Perspective for Uncertainty Quantification in Generative Models (NeurIPS 2025 · similarity 0.85)