Research map / Probabilistic and scientific modeling / Variational and generative sampling
Kernelised Normalising Flows
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Ferumal flows replace the neural-network scale and translation functions in coupling-layer normalising flows (RealNVP, Glow) with kernel expansions over data points, using auxiliary points for efficiency.
Problem
Normalising flows are over-parameterised neural networks that generalise poorly in low-data regimes, and parameter efficiency of flows is largely neglected.
Approach
Kernelise affine coupling layers: the scale and translation functions become representer-theorem kernel expansions over training inputs (k(u, data) weighted by learned coefficients), with learnable auxiliary points to reduce parameters; the parameter count then scales with dataset size rather than network width.
Key results
- Kernelised RealNVP/Glow reach up to 93% fewer parameters than neural counterparts (e.g. Power: 16K vs 228K) while matching or beating them on most tabular datasets (e.g. FF-Glow Gas 10.75 vs Glow 8.15 nats).
- In a 500-example low-data setting, the kernelised model beats FFJORD on all five datasets (e.g. Gas 0.22 vs -7.50 nats, BSDS300 121.22 vs 100.32), with far fewer parameters.
- On a Kuzushiji-MNIST VAE prior/posterior, FF-Glow gives ELBO -189.48 vs Glow -189.99, with negligible quantitative gain.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Kernelise RealNVP coupling layers | RealNVP · Power | log-likelihood: 0.17 -> 0.24 | Helped |
| Kernelise Glow coupling layers | Glow · Gas | log-likelihood: 8.15 -> 10.75 | Helped |
| Replace neural-network transformations with kernels | Flow models · Line | log-likelihood: 3.15 -> 3.75 | Helped |
| Replace neural-network transformations with kernels | Flow models · Pinwheel | log-likelihood: -2.48 -> -2.44 | Helped |
| Replace neural-network transformations with kernels | Flow models · Moons | log-likelihood: -2.54 -> -2.43 | Helped |
| Kernelise RealNVP and use shared auxiliary points | RealNVP · Miniboone, 500 examples | log-likelihood: -39.92 -> -27.75 | Helped |
| Kernelise RealNVP and use shared auxiliary points | RealNVP · Hepmass, 500 examples | log-likelihood: -28.17 -> -27.90 | Helped |
| Kernelise RealNVP and use shared auxiliary points | RealNVP · Gas, 500 examples | log-likelihood: -7.50 -> 0.22 | Helped |
| Kernelise RealNVP and use shared auxiliary points | RealNVP · Power, 500 examples | log-likelihood: -11.33 -> -2.91 | Helped |
| Kernelise RealNVP and use shared auxiliary points | RealNVP · BSDS300, 500 examples | log-likelihood: 100.32 -> 121.22 | Helped |
| Remove auxiliary points (ablation) | Ferumal flow · Miniboone, 500 examples | log-likelihood: -27.75 -> -41.83 | Hurt |
| Remove auxiliary points (ablation) | Ferumal flow · Hepmass, 500 examples | log-likelihood: -27.90 -> -29.01 | Hurt |
Surprising findings
- Neural-network RealNVP and Glow fail to generalise on 500-example subsets, while the kernelised version remains reasonable.
- Learned kernel length scale strongly affects overfitting versus underfitting, unlike neural-net flows where overfitting is controlled by adding layers or nodes.
Limitations
- Kernel choice and length scale are sensitive hyperparameters and imply inductive biases that must be selected.
- Method currently applies only to affine coupling architectures; not extended to ResFlow-type models.
- Parameter count still grows linearly with dataset size; image-scale application needs low-rank approximations explored only in appendix.
Experimental setup
Datasets: POWER GAS HEPMASS MINIBOONE BSDS300 UK Biobank CIFAR-10 Kuzushiji-MNIST 2D toy datasets
Benchmarks: POWER GAS HEPMASS MINIBOONE BSDS300
Metrics: log-likelihood (nats) parameter count training time
Baselines: RealNVP Glow MAF MADE FFJORD GF RBIG GIS
Builds on: RealNVP Glow FFJORD Gaussianisation Flows
Closest papers by meaning
- OT-Flow: Fast and Accurate Continuous Normalizing Flows via Optimal Transport (AAAI 2021 · similarity 0.88)
- CDFlow: Building Invertible Layers with Circulant and Diagonal Matrices (NeurIPS 2025 · similarity 0.87)
- Densely connected normalizing flows (NeurIPS 2021 · similarity 0.87)
- Self Normalizing Flows (ICML 2021 · similarity 0.87)
- Fast and unified path gradient estimators for normalizing flows (ICLR 2024 · similarity 0.85)
- Normalizing Flows are Capable Generative Models (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.