Research map / Large language models / AI and society

Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AI

ICML 2024 · Large language models › AI and society · approach: Benchmarking and reproducibility practice

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

A position paper arguing that Bayesian deep learning (BDL) can address overlooked needs of large-scale AI (uncertainty, data efficiency, continual/active learning, scientific data) and outlining challenges and research directions for scalable BDL.

Problem

Deep learning research emphasizes predictive accuracy on large supervised image and language tasks, neglecting uncertainty quantification, active and continual learning, and scientific data; BDL is seen as underused, mainly due to scalability concerns.

Approach

Argues that BDL's advantages (uncertainty quantification, prior incorporation, data efficiency, model averaging) can be made practical for foundation models by revisiting approximate inference (Laplace, variational, ensembles, SG-MCMC, function-space priors), and proposes future directions such as hybrid Bayesian approaches, deep kernel machines, and compression.

Key results

• No new experiments; the paper is a position and review piece with no quantitative results of its own.
• Reports that BNNs can achieve up to four times greater memory efficiency than deterministic counterparts in parameter count, citing prior work.
• Notes that HMC often outperforms deep ensembles but with significant additional computational overhead, citing prior work.

Surprising findings

• Argues that deep ensembles can be sensible approximate Bayesian inference when many functionally diverse modes exist, challenging the binary view of 'Bayesian' vs 'non-Bayesian'.

Limitations

• No new empirical validation of the proposed directions
• Scalability of BDL to LLM-scale models remains unresolved, as acknowledged by the authors

Experimental setup

Baselines: deep ensembles conformal prediction SWAG

Builds on: Laplace approximation deep ensembles stochastic gradient Langevin dynamics Stein variational gradient descent deep kernel learning

Closest papers by meaning

• Streamlining Prediction in Bayesian Deep Learning (ICLR 2025 · similarity 0.87)
• Bayesian Low-Rank Learning (Bella): A Practical Approach to Bayesian Neural Networks (AAAI 2025 · similarity 0.86)
• Scalable Bayesian Learning with posteriors (ICLR 2025 · similarity 0.86)
• A Bayesian Approach to Data Point Selection (NeurIPS 2024 · similarity 0.85)
• Variational Bayesian Pseudo-Coreset (ICLR 2025 · similarity 0.85)
• A Symmetry-Aware Exploration of Bayesian Neural Network Posteriors (ICLR 2024 · similarity 0.84)