Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes

Sparse Inducing Points in Deep Gaussian Processes: Enhancing Modeling with Denoising Diffusion Variational Inference

ICML 2024 · Probabilistic and scientific modeling › Uncertainty and Gaussian processes · approach: Sparse variational Gaussian processes

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Denoising Diffusion Variational Inference (DDVI), which uses a denoising diffusion SDE with score matching and a bridge-process trick to derive an explicit ELBO for inducing-point posteriors in deep Gaussian processes.

Problem

Posterior inference over inducing variables in deep Gaussian processes is biased under mean-field variational inference (DSVI) and unstable under implicit adversarial methods (IPVI).

Approach

Parameterize the inducing-point posterior as the terminal state of a learned denoising diffusion process whose score network is trained by minimizing KL divergence between path measures, using a bridge process (initialized at a fixed Gaussian) to obtain a tractable explicit variational lower bound.

Key results

• On MNIST, DDVI reaches 98.84% test accuracy with 3 layers versus 97.58% for IPVI and 97.17% for DSVI.
• On CIFAR-10 with ResNet-20 features, DDVI reaches 95.56% accuracy with 4 layers, versus 93.52% for IPVI.
• On SUSY and HIGGS, DDVI achieves the highest AUC at every depth (e.g., 0.857 HIGGS at L=5 vs 0.852 IPVI).

Measured effects reported in the paper

Limitations

• Hyperparameters and denoising network configurations chosen by manual grid search per dataset.
• Compared against a limited set of baselines (DSVI, IPVI, SGHMC) with no comparison to other modern DGP inference methods reported in main results.
• Derivation relies on several approximations (Riemann sums or ODE solvers for integrals) and the paper gives limited analysis of bias.

Experimental setup

Datasets: UCI regression benchmarks YearMSD Airline MNIST Fashion-MNIST CIFAR-10 SUSY HIGGS Frey Faces

Benchmarks: UCI regression (RMSE, NLL) YearMSD Airline MNIST Fashion-MNIST CIFAR-10 SUSY HIGGS Frey Faces GPLVM reconstruction

Metrics: RMSE NLL accuracy AUC runtime per iteration

Baselines: DSVI IPVI SGHMC

Builds on: Doubly Stochastic VI (DSVI) Implicit Posterior Variational Inference (IPVI) denoising diffusion probabilistic models score-based SDE GPyTorch

Closest papers by meaning

• Diffusion Bridge Variational Inference for Deep Gaussian Processes (ICLR 2026 · similarity 0.93)
• Denoising Diffusion Variational Inference: Diffusion Models as Expressive Variational Posteriors (AAAI 2025 · similarity 0.90)
• O(d/T) Convergence Theory for Diffusion Probabilistic Models under Minimal Assumptions (ICLR 2025 · similarity 0.88)
• Noise Conditional Variational Score Distillation (ICML 2025 · similarity 0.88)
• A Variational Perspective on Solving Inverse Problems with Diffusion Models (ICLR 2024 · similarity 0.87)
• Bayesian Basis Function Approximation for Scalable Gaussian Process Priors in Deep Generative Models (ICML 2025 · similarity 0.87)