Research map / Probabilistic and scientific modeling / Variational and generative sampling
Gradient Variance Reveals Failure Modes in Flow-Based Generative Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes gradient variance of rectified flow / conditional flow matching training and shows deterministic (noiseless) interpolants lead to memorization of arbitrary training pairings, with small noise restoring generalization.
Problem
Rectified Flows with deterministic couplings and noiseless straight-line interpolants can memorize training pairings rather than learn transport structure, and the role of gradient variance in selecting vector fields is unclear.
Approach
Uses gradient variance of the Monte Carlo CFM loss to characterize which vector fields optimization favors in Gaussian-to-Gaussian transport, proves a finite-dataset "minimizer memorizes" result for deterministic pairings, and shows small interpolant noise breaks the memorization pathway.
Key results
- On CelebA with adversarial shuffled pairings, CFM with sigma=0 has 5K memorization L2 to shuffled of 28.57 vs 55.02 for sigma=0.05, while sigma=0.05 gives better generalization L2 to OT (34.25 vs 50.40).
- In simulated 1-Reflow on CelebA, sigma=0 gives 5K memorization L2 to generated of 8.63 vs 25.08 for sigma=0.05.
- On a 3D/50D Gaussian mixture, CFM(sigma=0.05) yields lower MMD and Sinkhorn distances to the true distribution than CFM(sigma=0) in generalization comparisons.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add noise to interpolants (σ=0.05) (ablation) | CFM · Gaussian mixture d=3 Gen | MMD: 0.0034 -> 0.0018 | Helped |
| add noise to interpolants (σ=0.05) (ablation) | CFM · Gaussian mixture d=50 Gen | MMD: 0.0021 -> 0.0020 | Helped |
| add noise to interpolants (σ=0.05) (ablation) | CFM · CelebA adversarial 5K | L2 to OT (generalization): 50.40 ± 16.73 -> 34.25 ± 7.54 | Helped |
| add noise to interpolants (σ=0.05) (ablation) | CFM · CelebA adversarial 5K | L2 to shuffled targets (memorization): 28.57 ± 5.49 -> 55.02 ± 16.51 | Hurt |
| add noise to interpolants (σ=0.05) (ablation) | CFM · CelebA adversarial 50K | L2 to OT (generalization): 46.78 ± 14.87 -> 30.05 ± 6.77 | Helped |
| add noise to interpolants (σ=0.05) (ablation) | CFM · CelebA adversarial 50K | L2 to shuffled targets (memorization): 45.98 ± 11.85 -> 56.48 ± 18.55 | Hurt |
| add noise to interpolants (σ=0.05) (ablation) | CFM after simulated 1-ReFlow · CelebA 5K | L2 to OT (generalization): 43.35 ± 14.21 -> 31.35 ± 7.38 | Helped |
| add noise to interpolants (σ=0.05) (ablation) | CFM after simulated 1-ReFlow · CelebA 5K | L2 to generated endpoints (memorization): 8.63 ± 1.76 -> 25.08 ± 8.59 | Hurt |
| add noise to interpolants (σ=0.05) (ablation) | CFM after simulated 1-ReFlow · CelebA 50K | L2 to OT (generalization): 38.16 ± 11.37 -> 32.54 ± 8.86 | Helped |
| add noise to interpolants (σ=0.05) (ablation) | CFM after simulated 1-ReFlow · CelebA 50K | L2 to generated endpoints (memorization): 12.76 ± 4.01 -> 16.29 ± 4.82 | Hurt |
Surprising findings
- Gradient variance does not peak at interpolant intersections (e.g., 180-degree rotation), contrary to a common belief in flow matching.
- A memorizing vector field exists even when training interpolants intersect, and integration effectively bypasses intersection points, reproducing exact training pairings.
Limitations
- Evaluation limited to Gaussians, Gaussian mixtures, and CelebA at one resolution.
- Does not propose a remedy beyond adding interpolant noise.
- Some proofs and claims (e.g., MLP inability, Reflow idempotence) rely on informal or incomplete arguments.
Experimental setup
Datasets: CelebA Gaussian mixture (synthetic)
Benchmarks: CelebA
Metrics: L2 error to OT targets L2 error to shuffled/generated targets MMD Sinkhorn distance log-likelihood FID
Baselines: CFM (sigma=0) Rectified Flow Schrödinger Bridge Matching
Builds on: Rectified Flow Conditional Flow Matching stochastic interpolants
Closest papers by meaning
- On the Relation between Rectified Flows and Optimal Transport (NeurIPS 2025 · similarity 0.87)
- On Divergence Measures for Training GFlowNets (NeurIPS 2024 · similarity 0.87)
- Why Diffusion Models Don’t Memorize: The Role of Implicit Dynamical Regularization in Training (NeurIPS 2025 · similarity 0.87)
- Optimal Flow Matching: Learning Straight Trajectories in Just One Step (NeurIPS 2024 · similarity 0.86)
- How to build a consistency model: Learning flow maps via self-distillation (NeurIPS 2025 · similarity 0.86)
- Analysis of Learning a Flow-based Generative Model from Limited Sample Complexity (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.