Research map / Learning theory and algorithms / Sample complexity

Statistically Optimal Generative Modeling with Maximum Deviation from the Empirical Distribution

ICML 2024 · Learning theory and algorithms › Sample complexity · approach: Minimax estimation rates · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes LIPERM, a WGAN variant penalized by a left-inverse penalty on the generator, and proves that generators with smooth left inverses avoid replicating training data (W1 lower bound) while retaining minimax-optimal rates (finite-sample upper bound).

Problem

Generative models are usually analyzed for precision (closeness to the true distribution), but not for creativity, i.e., how far the generated distribution stays from the empirical training distribution (non-replication of training examples).

Approach

Penalize the push-forward generator g of the uniform distribution by the existence of a smooth left inverse h (h∘g ≈ Id), added to a WGAN objective; this forces a W1 separation from any distribution on the training points of order n^{-1/d} while keeping the n^{-1/d} statistical rate.

Key results

• Theorem lower bound: W1 between the learned generator and the empirical distribution is at least 1/(2L_H(1+(2V_d n)^{1/d})), i.e., order n^{-1/d}, for hard-constrained generators.
• Upper bound: E[d_F(generator, P*)] is bounded by the best penalized oracle plus 4σ* + ε + cL*sqrt(d)/n^{1/d}, with d>2, independent of ambient dimension D.
• On MNIST and CIFAR-10, lambda in {1,4,8} gives image quality comparable to vanilla WGAN (lambda=0), and the left-inverse penalty is minimized during training.

Measured effects reported in the paper

Surprising findings

• The experiments did not show increased creativity (non-replication) from the left-inverse penalty, since standard architectures did not replicate training examples in the first place.
• The penalty did not alleviate the known instability of WGAN training on Swiss Roll.

Limitations

• Computational difficulty of enforcing left invertibility; optimization errors likely dominant.
• Choice of latent dimension d not addressed; interplay with Lipschitz constant L* unclear.
• Analysis does not extend to diffusion models, higher-smoothness classes, or non-IPM distances such as Sinkhorn divergence.

Experimental setup

Datasets: Swiss Roll MNIST CIFAR-10

Benchmarks: Swiss Roll MNIST CIFAR-10

Metrics: Inception Score generator loss left-inverse penalty

Baselines: WGAN (lambda=0)

Builds on: Wasserstein GAN gradient penalty WGAN variational autoencoders CycleGAN

Closest papers by meaning

• Generative Modeling Using the Sliced Wasserstein Distance (CVPR 2018 · similarity 0.91)
• Entropic GANs meet VAEs: A Statistical Approach to Compute Sample Likelihoods in GANs (ICML 2019 · similarity 0.90)
• A Unifying View of Variational Generative Wasserstein Flows (ICML 2026 · similarity 0.90)
• Generative Modeling by Estimating Gradients of the Data Distribution (NeurIPS 2019 · similarity 0.89)
• Variational Annealing of GANs: A Langevin Perspective (ICML 2019 · similarity 0.89)
• Asymptotic Guarantees for Generative Modeling Based on the Smooth Wasserstein Distance (NeurIPS 2020 · similarity 0.89)