Research map / Probabilistic and scientific modeling / Variational and generative sampling
$t^3$-Variational Autoencoder: Learning Heavy-tailed Data with Student's t and Power Divergence
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes t^3VAE, a VAE with Student's t prior, encoder and decoder trained by minimizing a gamma-power divergence (gamma-loss) between joint model and data manifolds, which gives a power-form joint model.
Problem
Standard VAEs with Gaussian priors over-regularize and fail to capture heavy-tailed structure and low-density regions of data, particularly rare classes and tails.
Approach
Replace the ELBO's KL divergence between joint distributions with a gamma-power divergence and use Student's t-distributions for prior, encoder and decoder, yielding a closed-form gamma-loss whose degrees of freedom nu controls regularization strength (gamma = -2/(nu+n+m)).
Key results
- On CelebA reconstruction, t^3VAE (nu=10) achieves overall FID 39.4 versus 57.9 for the Gaussian VAE and 40.4 for beta-VAE (beta=0.05).
- On CIFAR100-LT with imbalance factor 100, t^3VAE reaches reconstruction FID 128.7 versus 287.3 for the VAE and 160.6 for beta-VAE.
- On the bivariate synthetic heavy-tailed data, t^3VAE is the only model whose MMD test does not reject in both left and right tails (p=0.214 and 0.213).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use t³VAE instead of β-VAE | VAE reconstruction on CelebA · CelebA reconstruction, all | FID: 40.4 -> 39.4 | Helped |
| Use t³VAE instead of Tilted VAE | VAE reconstruction on CIFAR100-LT · CIFAR100-LT reconstruction, ρ=100 | FID: 193.2 -> 128.7 | Helped |
| Use t³VAE instead of Gaussian VAE | VAE generation on univariate synthetic heavy-tailed data · Univariate synthetic heavy-tailed data, left and right tails | MMD p-value: 0.036 -> 0.377; 0.003 -> 0.693 | Helped |
| Use t³VAE instead of Student-t VAE | VAE generation on bivariate synthetic heavy-tailed data · Bivariate synthetic heavy-tailed data, left and right tails | MMD p-value: <0.001 -> 0.214; <0.001 -> 0.213 | Helped |
| Increase ν from moderate values toward very small values (ablation) | t³VAE · Low-dimensional synthetic data | generation quality | Hurt |
| Increase ν beyond the best range (ablation) | t³VAE · Univariate synthetic heavy-tailed data | tail density fit | Hurt |
| Replace the t³VAE prior with a higher-variance Gaussian prior (ablation) | VAE reconstruction on CelebA · CelebA reconstruction, all | FID: 39.4 -> 73.2 | Hurt |
Surprising findings
- Gaussian VAE with increased prior variance (kappa=1.5) performs worse than the standard VAE, indicating heavy-tailedness rather than simply larger variance drives gains.
- The original gamma-power divergence definition of prior work is non-negative only for gamma>0, requiring a modification.
Limitations
- Low-dimensional regimes with very small nu perform significantly worse; nu near 2 is problematic in low dimensions.
- The gamma-loss is an approximation valid to order gamma^2 and does not lower bound data likelihood.
Experimental setup
Datasets: CelebA CIFAR100-LT synthetic univariate heavy-tailed bimodal data synthetic bivariate heavy-tailed data
Benchmarks: CelebA CIFAR100-LT synthetic heavy-tailed bimodal data
Metrics: FID MMD test p-value
Baselines: VAE beta-VAE Student-t VAE DE-VAE VAE-st Tilted VAE FactorVAE VAE with increased prior variance HVAE
Builds on: variational autoencoder beta-VAE information geometry em-algorithm gamma-power divergence
Closest papers by meaning
- Phase-Type Variational Autoencoders for Heavy-Tailed Data (ICML 2026 · similarity 0.91)
- Variational Autoencoder with Implicit Optimal Priors (AAAI 2019 · similarity 0.90)
- Consistency Regularization for Variational Auto-Encoders (NeurIPS 2021 · similarity 0.89)
- Shape your Space: A Gaussian Mixture Regularization Approach to Deterministic Autoencoders (NeurIPS 2021 · similarity 0.89)
- Beyond Vanilla Variational Autoencoders: Detecting Posterior Collapse in Conditional and Hierarchical Variational Autoencoders (ICLR 2024 · similarity 0.88)
- A Statistical Analysis of Wasserstein Autoencoders for Intrinsically Low-dimensional Data (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.