Research map / Probabilistic and scientific modeling / Variational and generative sampling
Lifting Architectural Constraints of Injective Flows
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes free-form injective flows (FIF), which train an unconstrained encoder-decoder bottleneck with a fast single-step maximum-likelihood gradient estimator and an off-manifold encoder-Jacobian correction that prevents degenerate high-curvature manifold solutions.
Problem
Injective flows that jointly learn a low-dimensional data manifold and a likelihood on it are limited by restrictive invertible architectures and by the expense of estimating the change-of-variables log-determinant across dimensions; naive joint maximum-likelihood training is also ill-defined and ca
Approach
Replace the iterative conjugate-gradient estimator of rectangular flows with a single-step surrogate that uses the encoder Jacobian as the inverse of the decoder Jacobian, allowing free-form encoder/decoder networks, and evaluate the encoder Jacobian at x rather than the projection x̂ to counter the curvature-driven divergence.
Key results
- On tabular data FIF beats rectangular flows on POWER, HEPMASS and MINIBOONE in FID-like metric (e.g., HEPMASS 0.541 vs 0.779) with 1.5x-6.1x training speedup.
- On CelebA, FIF reaches FID 47.3 with N sampler and 37.4 with GMM sampler, versus 55.6/52.7 for DNF under equal compute.
- On the Pythae CelebA benchmark with ResNet and GMM sampling, FIF achieves FID 55.0, the best among compared generative autoencoders.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace RF with FIF | Tabular density estimation · POWER | FID-like metric: 0.083 $\pm$ 0.015 -> 0.041 $\pm$ 0.007 | Helped |
| Replace RF with FIF | Tabular density estimation · GAS | FID-like metric: 0.110 $\pm$ 0.021 -> 0.281 $\pm$ 0.031 | Hurt |
| Replace RF with FIF | Tabular density estimation · HEPMASS | FID-like metric: 0.779 $\pm$ 0.191 -> 0.541 $\pm$ 0.034 | Helped |
| Replace RF with FIF | Tabular density estimation · MINIBOONE | FID-like metric: 1.001 $\pm$ 0.051 -> 0.598 $\pm$ 0.024 | Helped |
| Use FIF instead of RF | Tabular density estimation · POWER, GAS, HEPMASS, MINIBOONE | Training time speedup: 3.9 $\times$, 2.2 $\times$, 6.1 $\times$, 1.5 $\times$ | Helped |
| Replace DNF with FIF | Injective flow image generation · CelebA, normal sampler | FID: 55.6 $\pm$ 0.59 -> 47.3 $\pm$ 1.39 | Helped |
| Replace Trumpet with FIF | Injective flow image generation · CelebA, GMM sampler | FID: 47.7 $\pm$ 2.24 -> 37.4 $\pm$ 1.35 | Helped |
| Use FIF with ResNet rather than the strongest reported baseline | Pythae image generation benchmark · CelebA, ResNet + normal sampler | FID: 64.8 -> 62.3 | Helped |
| Use FIF rather than VQVAE | Pythae image generation benchmark · CelebA, ConvNet + GMM sampler | FID: 51.6 -> 47.3 | Helped |
| Use FIF rather than Vanilla AE | Pythae image generation benchmark · CelebA, ResNet + GMM sampler | FID: 57.4 -> 55.0 | Helped |
| Use GMM latent sampling instead of standard normal sampling | FIF · CelebA image generation | FID: 56.9 -> 47.3; 62.3 -> 55.0 | Helped |
| Use the off-manifold encoder Jacobian rather than the on-manifold encoder Jacobian (ablation) | Free-form architecture · Tabular data | FID-like metric | Helped |
Surprising findings
- Naive maximum likelihood training of a bottleneck model is ill-defined: the loss is bounded below by the entropy of the projected data, and the model can collapse the manifold to lower entropy, including to a single point for linear models.
- Without the off-manifold encoder-Jacobian correction, the on-manifold estimator diverges for free-form architectures, while coupling-based injective architectures remain stable.
Limitations
- Estimator is only accurate when encoder and decoder are approximately reconstruction-optimal (pseudoinverse condition), which is assumed rather than guaranteed.
- Latent distributions are not perfectly Gaussian, so fitting a GMM after training improves image results.
- On GAS the method underperforms rectangular flows in the FID-like metric.
Experimental setup
Datasets: POWER GAS HEPMASS MINIBOONE CelebA MNIST CIFAR10
Benchmarks: POWER GAS HEPMASS MINIBOONE CelebA Pythae benchmark (MNIST, CIFAR10, CelebA)
Metrics: FID-like Wasserstein-2 metric FID Inception Score training time speedup
Baselines: Rectangular flows DNF Trumpet VAE IWAE FactorVAE WAE VQVAE RAE AAE
Builds on: rectangular flows injective flows Hutchinson trace estimator Pythae benchmark
Closest papers by meaning
- Canonical normalizing flows for manifold learning (NeurIPS 2023 · similarity 0.88)
- Rectangular Flows for Manifold Learning (NeurIPS 2021 · similarity 0.88)
- Injective flows for star-like manifolds (ICLR 2025 · similarity 0.86)
- Categorical Flow Matching on Statistical Manifolds (NeurIPS 2024 · similarity 0.85)
- Learning Distributions on Manifolds with Free-Form Flows (NeurIPS 2024 · similarity 0.85)
- Fast and unified path gradient estimators for normalizing flows (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.