Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes

Deep Kernel Posterior Learning under Infinite Variance Prior Weights

ICLR 2025 · Probabilistic and scientific modeling › Uncertainty and Gaussian processes · approach: Infinite-width networks as Gaussian processes · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that infinitely wide deep BNNs with elliptical infinite-variance (alpha-stable) weight priors converge to a deep kernel process with conditionally Gaussian stochastic kernels, and uses MCMC for posterior inference that enables feature learning.

Problem

Infinite-width limits of Bayesian neural networks with bounded-variance weights yield deterministic GP kernels, which preclude representation learning and cannot model jump discontinuities; existing infinite-variance (stable) posterior inference scales exponentially in input dimension and is limited

Approach

Use infinite-variance elliptical alpha-stable weight priors so each layer's limit is a conditionally Gaussian process with random scale mixing (alpha/2-stable), giving a stochastic covariance kernel recursively linked across layers (Cho & Saul-style arc-cosine recursion), with computation in kernel space rather than feature space.

Key results

• On synthetic 1D discontinuous regression D-alpha-KP achieved RMSE 0.57 vs 1.08 for DIWP and 0.52 for Stable (best), and in 10D achieved best RMSE 8.08 vs 8.32 for GP MLE.
• On UCI Energy and Yacht, D-alpha-KP achieved the lowest RMSE (0.46 and 0.31), while on Boston it was close second (2.59 vs 2.58 for GP Bayes).
• Coverage of 90% credible intervals was about 90% in simulations (89.70%-92.47%), whereas DIWP and NNGP reached only about 60% in 10D.

Measured effects reported in the paper

Surprising findings

• Depth had no meaningful effect on predictive performance: L=3 to L=16 gave nearly identical errors on the synthetic datasets.
• The Stable method achieved slightly better RMSE than D-alpha-KP in 1D and 2D simulations, despite D-alpha-KP being much faster in 2D.

Limitations

• The Stable baseline and the exponential-complexity method could not be run beyond two dimensions, so high-dimensional comparisons are limited.
• Experiments use only shallow (L=2) settings for the main comparisons, and claimed benefits of depth are not demonstrated.
• Relies on MCMC, which is computationally heavy (running times of thousands of seconds on UCI data); variational or inducing-point approaches are left to future work.

Experimental setup

Datasets: synthetic discontinuous functions (1D, 2D, 10D) UCI Boston UCI Energy UCI Yacht

Benchmarks: UCI Boston UCI Energy UCI Yacht synthetic discontinuous regression 1D/2D/10D

Metrics: RMSE MAE coverage of 90% credible intervals running time

Baselines: DIWP NNGP GP Bayes GP MLE Stable

Builds on: Neal (1996) BNN-GP limit Cho & Saul (2009) deep arc-cosine kernels deep inverse Wishart process (DIWP) Stable process posterior inference (Stable method)

Closest papers by meaning

• Deep Neural Networks as Gaussian Processes (ICLR 2018 · similarity 0.90)
• A theory of representation learning gives a deep generalisation of kernel methods (ICML 2023 · similarity 0.88)
• Adaptive kernel predictors from feature-learning infinite limits of neural networks (ICML 2025 · similarity 0.88)
• Convolutional Deep Kernel Machines (ICLR 2024 · similarity 0.87)
• Critical feature learning in deep neural networks (ICML 2024 · similarity 0.87)
• Beyond IID weights: sparse and low-rank deep Neural Networks are also Gaussian Processes (ICLR 2024 · similarity 0.86)