Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes

Beyond IID weights: sparse and low-rank deep Neural Networks are also Gaussian Processes

ICLR 2024 · Probabilistic and scientific modeling › Uncertainty and Gaussian processes · approach: Infinite-width networks as Gaussian processes · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Extends the Gaussian process limit of random deep networks to "Pseudo-iid" weight distributions (exchangeable, uncorrelated entries) covering orthogonal, low-rank and structured sparse weights for fully connected and convolutional networks.

Problem

Gaussian process limit of infinitely wide random deep networks was previously proven only for iid weights (and orthogonal weights partially), excluding low-rank and structured sparse initializations used for efficiency.

Approach

Relaxes the iid requirement to row- and column-exchangeable, uncorrelated weights with moment conditions, proves a simultaneous-width GP limit via an exchangeable CLT, and extends the definition to convolutional kernels and orthogonal CNN filters.

Key results

• Numerical simulations of tanh fully connected networks with low-rank, block-sparse, dropout and orthogonal weights show histograms matching the predicted limiting variance already at width n=30 and improving at n=300.
• Orthogonal initialization shows the fastest convergence to the Gaussian process among the tested Pseudo-iid distributions.
• Pseudo-iid networks have Gaussian process limits with covariance kernels identical to iid ones up to variance rescaling (e.g., by rank or sparsity fraction), enabling exact NNGP inference and edge-of-chaos analysis.

Surprising findings

• Structured sparse or low-rank initializations behave like dense counterparts up to a fractional variance rescaling.

Limitations

• First-layer weights must be iid (or row-iid) rather than general Pseudo-iid
• Whether all Pseudo-iid conditions are necessary remains open
• Validation limited to modest widths (up to 300) in simulations

Experimental setup

Metrics: histogram agreement with predicted Gaussian variance Q-Q plots joint empirical distribution vs theoretical covariance

Builds on: Neal (1996) infinite-width GP Lee et al. 2018 deep GP Matthews et al. 2018 sequential-limit GP Pennington et al. orthogonal initialization Yang tensor programs

Closest papers by meaning

• Wide Neural Networks as Gaussian Processes: Lessons from Deep Equilibrium Models (NeurIPS 2023 · similarity 0.88)
• Bayesian Neural Network Priors Revisited (ICLR 2022 · similarity 0.87)
• DEEP NEURAL NETWORK INITIALIZATION WITH SPARSITY INDUCING ACTIVATIONS (ICLR 2024 · similarity 0.86)
• Emergence and scaling laws in SGD learning of shallow neural networks (NeurIPS 2025 · similarity 0.86)
• Deep Kernel Posterior Learning under Infinite Variance Prior Weights (ICLR 2025 · similarity 0.86)
• Feature Learning Dynamics in Infinite-Depth Neural Networks (NeurIPS 2026 · similarity 0.86)