Research map / Optimization / Optimizers
Optimization and Bayes: A Trade-off for Overparameterized Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace Hutchinson Hessian-trace estimation with a fixed-factor approximation derived from the dynamics analysis | Hessian trace estimation · Synthetic unit-circle data, 3-layer width-2048 tanh network | agreement with Hutchinson ground truth (Figure 1) | No clear effect |
| describe training in function space via the infinite-width dynamics instead of running SGD in parameter space | overparameterized MLP, MSE regression on 1/2 sin(3 tau) · Synthetic toy regression | output and Hessian-trace dynamics | No clear effect |
| use TransBL with weight clipping and large beta (closer to Bayesian posterior) | TransBL · Fashion-MNIST one-shot binary classification | test accuracy | Helped |
| use TransBL with weight clipping and large beta (closer to Bayesian posterior) | TransBL · Fashion-MNIST one-shot binary classification | sampling efficiency (effective samples) | Hurt |
| use small beta in TransBL weight clipping (behaves like deep ensemble) | TransBL · Fashion-MNIST one-shot binary classification | test accuracy | Hurt |
| vary beta in weight clipping: 0.2 vs 1 vs 5 | TransBL on univariate two-minima example · Illustrative example (Figure 7) | closeness of ensemble to Bayesian posterior | Helped |
| apply importance weight to gradient-descent ensemble (TransBL) instead of plain gradient flow | gradient descent ensemble · Univariate loss with sharp and flat global minima | match to Bayesian posterior | Helped |
Closest papers by meaning
- Bayesian Optimization with Robust Bayesian Neural Networks (NeurIPS 2016 · similarity 0.88)
- Bayesian Optimization with Gradients (NeurIPS 2017 · similarity 0.88)
- Bayesian filtering unifies adaptive and non-adaptive neural network optimization methods (NeurIPS 2020 · similarity 0.88)
- A Study of Bayesian Neural Network Surrogates for Bayesian Optimization (ICLR 2024 · similarity 0.88)
- Bayesian Online Natural Gradient (BONG) (NeurIPS 2024 · similarity 0.87)
- Bayesian Optimization via Continual Variational Last Layer Training (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.