Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes

Exact Gaussian Moment Matching for Residual Networks: a Second-Order Method

NeurIPS 2026 · Probabilistic and scientific modeling › Uncertainty and Gaussian processes · approach: Variational Bayesian deep learning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Derives exact closed-form Gaussian moment matching (mean and covariance) for probit, GeLU, ReLU, Heaviside and sine activations in feedforward and residual layers, proves a fourth-order smooth-distance error bound, and shows large KL improvements over linearization, unscented and mean-field baselines.

Problem

Propagating the mean and covariance of a multivariate Gaussian through deep (residual) networks layer by layer; existing moment-matching methods restrict input means/covariances, approximate activations, or omit residual layers.

Approach

Exact closed-form expressions for the Gaussian moments of activation outputs (including cross-covariances with pre-activations for residual layers), derived using Stein's identity and Gaussian ODEs without R^2 numerical integration, plus a fourth-order smooth-distance theorem showing layer-wise moment matching is second-order accurate.

Key results

• On random networks with large input variance, the method is about one hundred times better than Unscented'95, ten thousand times better than linearization, and one million times better than Unscented'02 by KL divergence from the pseudo-true distribution.
• On variational Bayes neural networks, exact moment matching gives the lowest KL divergence from Monte Carlo predictive distributions, hundredfold better than the state-of-the-art deterministic method (power series expansion) in the abstract's claim.

Surprising findings

• Moment matching is #P-hard in general, so exactness for single hidden layers does not extend to all multi-layer networks; a constructed Heaviside example makes the method arbitrarily wrong.

Limitations

• Does not apply to softmax or attention.
• Not tested on large-scale problems or benchmarked against applied ML methods.
• Multi-layer moment matching is not exact in general, and can fail on constructed strongly non-normal, nonlinear networks.

Experimental setup

Datasets: UCI regression datasets (eleven)

Benchmarks: random neural network ensembles UCI regression datasets (variational Bayes)

Metrics: KL divergence from pseudo-true Gaussian Wasserstein distance to true distribution

Baselines: linearization (delta method) Unscented'95 Unscented'02 mean-field approximation power series expansion (W24)

Builds on: Gaussian moment matching unscented transformation mean-field approximation delta method linearization power series activation moment approximation

Closest papers by meaning

• Structured and Efficient Variational Deep Learning with Matrix Gaussian Posteriors (ICML 2016 · similarity 0.87)
• Conditional Generative Moment-Matching Networks (NeurIPS 2016 · similarity 0.87)
• Efficient Low Rank Gaussian Variational Inference for Neural Networks (NeurIPS 2020 · similarity 0.86)
• Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes (ICLR 2019 · similarity 0.86)
• Deterministic equivalent and error universality of deep random features learning (ICML 2023 · similarity 0.86)
• Inductive Moment Matching (ICML 2025 · similarity 0.85)