Research map / Optimization / Optimizers
Debiasing Mini-Batch Quadratics for Applications in Deep Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that curvature and slope of mini-batch quadratics are systematically biased (curvature overestimated ~10x along top eigendirections), explains it via regression to the mean/misaligned eigenspaces, and proposes two-batch debiasing for conjugate gradients and K-FAC Laplace approximation.
Problem
Mini-batch estimates of the loss's local quadratic approximation are biased in shape, which harms second-order optimizers (Newton/CG steps) and Laplace-approximation uncertainty in deep learning.
Approach
Decouple the subspace (directions from one mini-batch) from the slope and curvature measurements, which are re-estimated on a different mini-batch; this debiases CG update magnitudes and K-FAC Laplace covariance eigenvalues, explained as regression to the mean caused by misaligned eigenspaces.
Key results
- Mini-batch curvature along top eigenvectors is overestimated by roughly one order of magnitude when eigenvectors and curvature are computed on the same mini-batch (All-CNN-C, CIFAR-100).
- Debiased CG remains stable without damping, while single-batch CG diverges quickly after peak performance, at similar computational cost.
- Debiased K-FAC Laplace approximation achieves good performance over a wider range of prior precisions and mimics the full-batch LA (N=40000) on CIFAR-10 while using a tiny fraction of the data.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a second mini-batch to estimate CG update magnitudes while retaining directions from the first batch | CG on All-CNN-C · CIFAR-100 | stability | Helped |
| Use one mini-batch for both CG directions and update magnitudes instead of estimating magnitudes on a second batch | CG on All-CNN-C · CIFAR-100 | stability | Hurt |
| Use single-batch CG rather than debiased CG | CG on All-CNN-C · CIFAR-100 | peak performance | Helped |
| Use a second mini-batch to estimate directional curvatures while retaining the eigenspace from the first batch | KFAC Laplace on All-CNN-C · CIFAR-10 | performance across prior precisions | Helped |
| Use one mini-batch for both KFAC directions and curvatures rather than estimating curvatures on a second batch | KFAC Laplace on All-CNN-C · CIFAR-10 | performance at low prior precision | Hurt |
| Increase mini-batch size from 512 to 2048 (ablation) | Mini-batch quadratic curvature estimates · CIFAR-100 | curvature bias | Helped |
| Use the Hessian instead of its GGN approximation (ablation) | Mini-batch quadratic curvature estimates · CIFAR-100 | curvature bias | No clear effect |
| Use CG search directions instead of eigenvectors (ablation) | Mini-batch quadratic slope estimates · CIFAR-100 | directional slope bias | Hurt |
| Compute eigenvectors and directional curvatures on the same mini-batch instead of separate batches | Mini-batch quadratic · CIFAR-100 | directional curvature | Hurt |
Surprising findings
- Mini-batch slopes along CG directions are predominantly negative while full-batch slopes are positive for most CG directions.
- Single-batch CG divergence is attributed to misinformed update magnitudes rather than missing damping.
Limitations
- Debiasing roughly doubles per-step computation (second mini-batch needed).
- Peak performance of debiased CG is slightly below single-batch CG, which uses double the data for directions.
- Analysis for K-FAC does not follow the same unbiasedness argument since averaged K-FAC is not the K-FAC of the union.
Experimental setup
Datasets: CIFAR-100 CIFAR-10 ImageNet
Benchmarks: CIFAR-100 CIFAR-10 ImageNet
Metrics: training loss test loss training accuracy test accuracy NLL ECE
Baselines: single-batch CG single-batch K-FAC Laplace approximation full-batch Laplace approximation vanilla model without LA
Builds on: K-FAC Laplace approximation conjugate gradients EKFAC
Closest papers by meaning
- On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima (ICLR 2017 · similarity 0.86)
- Shuffling the Data, Extrapolating the Step: Sharper Bias In Constant Step-Size SGD (ICLR 2026 · similarity 0.86)
- A Scalable Laplace Approximation for Neural Networks (ICLR 2018 · similarity 0.86)
- FSP-Laplace: Function-Space Priors for the Laplace Approximation in Bayesian Deep Learning (NeurIPS 2024 · similarity 0.85)
- Non-asymptotic Analysis of Biased Adaptive Stochastic Approximation (NeurIPS 2024 · similarity 0.84)
- Loss Landscape Characterization of Neural Networks without Over-Parametrization (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.