Research map / Optimization / Implicit bias and generalization

Estimating Implicit Regularization in Deep Learning

NeurIPS 2026 · Optimization › Implicit bias and generalization · approach: Explicit and implicit regularization interplay · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes gradient-matching estimators that treat trained weights as an optimum of loss plus a parametric regularizer, recovering the regularization strength by regressing negative loss gradients onto regularizer gradients, and applies them to elastic net, early stopping in OLS, dropout, and discrete gradient steps.

Problem

Implicit regularization induced by complex deep learning procedures (early stopping, minibatching, dropout) is hard to derive analytically and has received little direct empirical estimation.

Approach

Assume the learned weights are an exact optimum of a regularized objective (or that weight-update deviations from the loss gradient are driven by a regularizer), and estimate regularizer parameters by least-squares matching of the regularizer gradients to the negative loss gradients, with a martingale trajectory extension and PAC-style guarantees.

Key results

• Elastic-net penalties on deep ReLU networks are recovered accurately for intermediate penalty values, with identifiability limits for very small and large penalties.
• In OLS, a scalar heuristic lambda_t estimated from early-stopped gradient descent decays toward zero over steps, matching theory closely; a diagonal underparameterized penalty reproduces the early-stopped weights exactly.
• On MNIST, estimated ridge penalty increases monotonically with dropout rate across depth and width settings.

Surprising findings

• An underparameterized diagonal implicit-regularization estimate, used as an explicit penalty without early stopping, reproduces the early-stopped weights exactly.

Limitations

• Number of fitted regularizer parameters is bounded by the parameter dimension p and effective sample size; collinear gradients inflate variance.
• Recovery of the original weights does not imply identification of the true regularizer.
• Assumes the learned weights are a (hypothetical) optimum of a regularized objective, which may be a poor approximation early in training.

Experimental setup

Datasets: MNIST synthetic linear regression data (1000x10)

Benchmarks: MNIST

Metrics: estimated regularization strength gradient-matching loss test accuracy estimation error vs theoretical value

Baselines: Ali et al. theoretical early-stopping regularizer

Builds on: implicit gradient regularization early stopping implicit regularization in OLS (Ali et al.) dropout as adaptive ridge

Closest papers by meaning

• Combining Explicit and Implicit Regularization for Efficient Learning in Deep Networks (NeurIPS 2022 · similarity 0.91)
• Mirror, Mirror of the Flow: How Does Regularization Shape Implicit Bias? (ICML 2025 · similarity 0.89)
• On the Implicit Bias of Adam (ICML 2024 · similarity 0.88)
• Implicit Regularization Paths of Weighted Neural Representations (NeurIPS 2024 · similarity 0.88)
• Implicit regularization of deep residual networks towards neural ODEs (ICLR 2024 · similarity 0.86)
• Mask in the Mirror: Implicit Sparsification (ICLR 2025 · similarity 0.86)