Research map / Architectures and efficiency / RNNs and normalization
CDFlow: Building Invertible Layers with Circulant and Diagonal Matrices
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CDFlow, a normalizing flow whose invertible 1x1 linear layers are products of diagonal and circulant matrices, enabling O(mn) log-determinant and O(mn log n) inversion via FFT.
Problem
Normalizing flows need invertible linear layers that are expressive yet cheap to invert and to compute log-determinants for; dense weight matrices cost O(n^3) for log-det and inverse.
Approach
Parameterize the invertible linear layer as an alternating product of m diagonal and m-1 circulant matrices (a DCD factorization), so the log-determinant is a sum of log-eigenvalues and the inverse uses FFT-diagonalized circulant products.
Key results
- CDFlow reaches 3.31 BPD on CIFAR-10 (44.2M params), beating Glow (3.36) and Emerging (3.34) at comparable size, though behind Residual Flows (3.28) and i-DenseNet (3.25).
- On Galaxy/structured images, CDFlow reports a 4.95% BPD improvement over Glow.
- Runtime: with 96 channels, CDFlow speeds up matrix inverse by 1.17x and log-determinant by 4.31x versus dense matrix.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use circulant-diagonal linear layers instead of Glow's dense linear layer | CDFlow · Galaxy | BPD: 4.95% improvement | Helped |
| use circulant-diagonal linear layers | CDFlow · CIFAR-10 | BPD: 3.25; $3.31 \pm 0.004$ | Mixed |
| use circulant-diagonal linear layers | CDFlow · ImageNet 32×32 | BPD: 3.98; 4.04 | Mixed |
| replace standard MLP linear layers with CD linear layers | Flow Matching model · Chessboard | NLL: 8.2088 -> 7.0425 | Helped |
| replace standard MLP linear layers with CD linear layers | Flow Matching model · Moons | MSE: 1.854 -> 1.842 | Helped |
| replace standard MLP linear layers with CD linear layers | Flow Matching model · Moons | FID: 0.0016 | No clear effect |
| replace standard MLP linear layers with CD linear layers | Flow Matching model · Circles | NLL: 20.525 -> 15.4922 | Helped |
| replace standard MLP linear layers with CD linear layers | Flow Matching model · Circles | MSE: 2.433 -> 2.482 | Hurt |
| replace standard MLP linear layers with CD linear layers | Flow Matching model · Circles | FID: 0.0019 -> 0.0069 | Hurt |
| increase the number of diagonal-circulant factors m (ablation) | CDFlow · CIFAR-10 | BPD | No clear effect |
| increase the number of diagonal-circulant factors m (ablation) | CDFlow · CIFAR-10 | parameter count | Hurt |
| use DCD-type rather than LU-type linear layers (ablation) | flow-based model · Appendix linear-layer comparison | NLL | Helped |
Surprising findings
- The DCD-type layer does not match the fully learnable F-type linear layer in NLL, though it beats the LU-type layer.
Limitations
- Currently restricted to 1x1 convolution; d x d extension left to future work.
- Does not beat the strongest reported baselines (Residual Flows, i-DenseNet) on CIFAR-10 BPD.
- Inverse is slightly less efficient than 1x1 convolution at large spatial sizes.
Experimental setup
Datasets: CIFAR-10 ImageNet-32x32 Galaxy CIFAR-100
Benchmarks: CIFAR-10 ImageNet-32x32 Galaxy CIFAR-100 2D toy datasets (Chessboard, Moons, Circles)
Metrics: bits per dimension NLL MSE FID runtime (ms)
Baselines: MAF Real NVP Glow Emerging Residual Flows i-ResNet Woodbury i-DenseNet ButterflyFlow ME-Woodbury Periodic convolution
Builds on: Glow Emerging convolution Woodbury transformation ButterflyFlow RealNVP
Closest papers by meaning
- Invertible Convolutional Flow (NeurIPS 2019 · similarity 0.89)
- Densely connected normalizing flows (NeurIPS 2021 · similarity 0.89)
- ButterflyFlow: Building Invertible Layers with Butterfly Matrices (ICML 2022 · similarity 0.89)
- Kernelised Normalising Flows (ICLR 2024 · similarity 0.87)
- Normalizing Flows are Capable Generative Models (ICML 2025 · similarity 0.86)
- STARFlow: Scaling Latent Normalizing Flows for High-resolution Image Synthesis (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.