Research map / Probabilistic and scientific modeling / Variational and generative sampling
Functional Gradient Flows for Constrained Sampling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes constrained functional gradient flow (CFG), a neural-network functional gradient ParVI with a piecewise boundary-aware velocity field and a band-wise Monte Carlo estimator for boundary integrals, for sampling from inequality-constrained domains with TV convergence guarantees.
Problem
Sampling from target distributions supported on constrained domains with general inequality constraints, where existing ParVI and MCMC methods are either kernel-based, expensive, or limited to simple geometries.
Approach
Extends neural functional gradient ParVI to constrained domains by using a piecewise velocity field (learned inside, normalized gradient of the constraint pushing outside particles inward) and estimates the extra boundary integral term from the regularized Stein discrepancy via a band-wise Monte Carlo approximation using particles near the boundary.
Key results
- On the monotonic BNN with epsilon=0.01 on COMPAS, CFG reaches test accuracy 0.660 vs 0.645 (PD-SVGD), 0.650 (C-SVGD), 0.590 (MIED), with 0.0% particles outside the domain.
- On toy double-moon domain, CFG reduces Sinkhorn Wasserstein-2 distance to 0.1660 from MIED's 0.4724 and energy distance to 0.0022 from 0.0629.
- CFG provides a TV convergence bound of O(T^{-1/2} + epsilon^{1/2}) for the minimum over time, under a Poincare inequality and neural approximation error assumption.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CFG instead of C-SVGD | Constrained monotonic Bayesian neural network · COMPAS, ε=0.05, 0.01, 0.005 | Test accuracy: $.649\pm .002$ -> $.661\pm .000$; $.650\pm .002$ -> $.660\pm .001$; $.650\pm .002$ -> $.659\pm .001$ | Helped |
| Use CFG instead of C-SVGD | Constrained monotonic Bayesian neural network · COMPAS, ε=0.05, 0.01, 0.005 | Test NLL: $.633\pm .001$ -> $.632\pm .000$; $.634\pm .001$ -> $.632\pm .000$; $.633\pm .002$ -> $.632\pm .000$ | Helped |
| Use CFG instead of MIED | Constrained sampling · Ring | Wasserstein-2 distance (Sinkhorn): 0.1074 -> 0.1087 | Hurt |
| Use CFG instead of MIED | Constrained sampling · Ring | Energy distance: 0.0004 -> 0.0003 | Helped |
| Use CFG instead of MIED | Constrained sampling · Cardioid | Wasserstein-2 distance (Sinkhorn): 0.1240 -> 0.1141 | Helped |
| Use CFG instead of MIED | Constrained sampling · Cardioid | Energy distance: 0.0016 -> 0.0005 | Helped |
| Use CFG instead of MIED | Constrained sampling · Double-moon | Wasserstein-2 distance (Sinkhorn): 0.4724 -> 0.1660 | Helped |
| Use CFG instead of MIED | Constrained sampling · Double-moon | Energy distance: 0.0629 -> 0.0022 | Helped |
| Add boundary integral estimation (ablation) | CFG · Ring, Cardioid, Double-moon, and Block | Wasserstein-2 distance (Sinkhorn): 0.2138 -> 0.1087; 0.2321 -> 0.1660; 0.4866 -> 0.1141; 0.2438 -> 0.2416 | Helped |
| Add z_net (ablation) | CFG · Ring, Cardioid, Double-moon, and Block | Wasserstein-2 distance (Sinkhorn): 0.1248 -> 0.1087; 0.2234 -> 0.1660; 0.1217 -> 0.1141; 0.2422 -> 0.2416 | Helped |
| Add z_net (ablation) | CFG · Ring, Cardioid, Double-moon, and Block | Energy distance: 0.0013 -> 0.0003; 0.0009 -> 0.0005; 0.0049 -> 0.0022; 0.0073 -> 0.0072 | Helped |
| Add boundary integral estimation (ablation) | CFG · Ring, Cardioid, Double-moon, and Block | Energy distance: 0.0097 -> 0.0003; 0.1147 -> 0.0005; 0.0068 -> 0.0022; 0.0073 -> 0.0072 | Helped |
Surprising findings
- Without estimating the boundary integral term, sampling fails in most toy cases (e.g., Ring W2 rises from 0.1087 to 0.2138), showing the boundary term is essential for constrained sampling.
- A globally continuous velocity field may fail to attain the RSD minimum under inequality constraints, motivating the piecewise design.
Limitations
- Assumes particles outside the domain do not get stuck at local extrema of g and requires the gradient norm lower bound outside the domain.
- Bandwidth selection for the boundary band requires tuning h0 by sparse grid search.
- Convergence analysis relies on Poincare inequality for the target and a neural approximation error assumption.
Experimental setup
Datasets: synthetic 20-dimensional regression dataset (1000x20) diabetes dataset COMPAS Blog Feedback
Benchmarks: Wasserstein-2 distance (Sinkhorn) energy distance COMPAS monotonic BNN test accuracy and NLL diabetes Bayesian Lasso and Bridge posterior medians
Metrics: Wasserstein-2 distance energy distance test accuracy test NLL ratio out
Baselines: MSVGD MIED Spherical HMC PD-SVGD Control SVGD
Builds on: Stein Variational Gradient Descent functional Wasserstein gradient regularized Stein discrepancy reflected stochastic differential equations
Closest papers by meaning
- Particle-based Variational Inference with Preconditioned Functional Gradient Flow (ICLR 2023 · similarity 0.93)
- Particle-based Variational Inference with Generalized Wasserstein Gradient Flow (NeurIPS 2023 · similarity 0.92)
- Particle Semi-Implicit Variational Inference (NeurIPS 2024 · similarity 0.90)
- Asymptotically exact variational flows via involutive MCMC kernels (NeurIPS 2025 · similarity 0.88)
- Posterior Sampling Based on Gradient Flows of the MMD with Negative Distance Kernel (ICLR 2024 · similarity 0.87)
- Kernel Semi-Implicit Variational Inference (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.