Research map / Probabilistic and scientific modeling / Variational and generative sampling
Flexible Tails for Normalizing Flows
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes the tail transform flow (TTF), a Gaussian-base normalizing flow with a final erfc-based non-Lipschitz transformation that produces tunable heavy tails, outperforming heavy-tailed base-distribution flows on density estimation and variational inference.
Problem
Standard normalizing flows with Gaussian bases and Lipschitz transformations cannot represent heavy tails, while existing fixes using heavy-tailed base distributions induce poorly conditioned neural network optimisation under heavy-tailed inputs.
Approach
Use a Gaussian base and a final monotone erfc-based transformation R(z) that maps Gaussian tails to generalized Pareto tails with separately tunable upper and lower tail weights, so the neural network body never sees extreme inputs.
Key results
- On synthetic d=50, nu=0.5, TTF achieves test NLL 3.68 versus 7.49 for gTAF and 5.22 for mTAF, while normal flow fails to converge.
- On CLIMDEX, TTF reaches test NLL -2214.28 vs -2121.38 for mTAF and -2101.91 for normal flow, the best of the compared methods.
- In variational inference on the artificial target, TTFfix attains the best ESS_e for heavier tails (nu<=2), and gTAF and mTAF are worse than TTF methods for nu<=1, d=50.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Model heavy tails in a final TTF transformation instead of using a heavy-tailed base | Normalizing flow density estimation · Synthetic density estimation, d=50, ν≤1 | negative test log likelihood per dimension: 3.74 (0.00) -> 3.68 (0.00); 2.55 (0.00) -> 2.54 (0.00) | Helped |
| Use TTF rather than COMET | Normalizing flow density estimation · Synthetic density estimation, d=50, ν≤1 | negative test log likelihood per dimension: 3.74 (0.00) -> 3.68 (0.00); 2.55 (0.00) -> 2.54 (0.00) | Helped |
| Model tails in the final transformation | Normalizing flow density estimation · Synthetic density estimation, d=50, ν=2 | negative test log likelihood per dimension: 1.98 (0.00) vs 1.97 (0.00) | No clear effect |
| Use TTF rather than TTFfix, allowing tail parameters to be learned (ablation) | TTF density estimation · Insurance, Fama 5, S&P 500, CLIMDEX | mean test negative log likelihood: 1.38 (0.01) -> 1.37 (0.02); 4.63 (0.01) -> 4.61 (0.01); 314.84 (0.46) -> 317.56 (0.56); -2090.91 (11.90) -> -2214.28 (13 | Mixed |
| Add TTF as a final transformation | Normalizing flow density estimation · Insurance, Fama 5, S&P 500, CLIMDEX | mean test negative log likelihood | Helped |
| Fix tail parameters instead of learning them jointly (ablation) | TTF variational inference · Artificial target VI, ν≤2 | ESS_e: 0.39 -> 0.43; 0.57 -> 0.75; 0.61 -> 0.81 | Helped |
| Use a final tail transformation rather than a Student's T base distribution | Variational inference · Artificial target VI, ν≤1, d=50 | ESS_e | Helped |
| Fix tail parameters rather than train them (ablation) | Tail-adaptive flow density estimation · Synthetic density estimation, d=50, ν≤1 | negative test log likelihood per dimension: 5.22 (0.04) vs 7.49 (0.38); 2.62 (0.01) vs 2.65 (0.01) | Helped |
| Combine TTF with a trainable Student's T base (ablation) | TTF density estimation · Synthetic density estimation | negative test log likelihood per dimension | Hurt |
| Use an autoregressive structure to generate marginal tail parameters (ablation) | TTF · Density estimation | optimization performance | Hurt |
Surprising findings
- COMET flows perform competitively despite theory showing they cannot produce Frechet-domain tails, possibly because log-normal tails resemble GPD tails sub-asymptotically.
- Gaussian mixture base distributions remain light tailed, so they do not solve the heavy-tail problem.
Limitations
- The transformation always converts Gaussian tails to heavy tails, so exactly Gaussian tails cannot be produced.
- The TTF transformation affects both body and tail, so preceding layers must adapt to the final layer.
- In real VI applications tail parameters are unknown and difficult to estimate; the VI gains are shown only on an artificial target.
Experimental setup
Datasets: synthetic heavy-tailed t-distribution dependence data Insurance Fama 5 S&P 500 CLIMDEX
Benchmarks: Insurance Fama 5 S&P 500 CLIMDEX
Metrics: negative test log likelihood per dimension negative test log likelihood ESS_e k-hat (Pareto diagnostic)
Baselines: normal flow mTAF gTAF COMET m_normal g_normal TTF_tBase
Builds on: tail adaptive flows COMET flows neural autoregressive flows rational quadratic spline flows
Closest papers by meaning
- Heavy-Tailed Diffusion Models (ICLR 2025 · similarity 0.86)
- Heavy-Tailed Flow Matching via Random Clocks (NeurIPS 2026 · similarity 0.85)
- Normalizing Flows are Capable Generative Models (ICML 2025 · similarity 0.85)
- Emergence of heavy tails in homogenized stochastic gradient descent (NeurIPS 2024 · similarity 0.84)
- On the Importance of Gaussianizing Representations (ICML 2025 · similarity 0.83)
- Kernelised Normalising Flows (ICLR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.