Research map / Learning theory and algorithms / Kernels and regression theory
Implicit Regularization Paths of Weighted Neural Representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives implicit regularization paths (equivalences between weighted ridge estimators on pretrained features and explicit ridge on full data) via free probability, extends to random/kernel features and ensembles, and uses them for an efficient cross-validation tuning method.
Problem
Understanding how observation weighting (notably subsampling without replacement) of pretrained neural features acts as implicit ridge regularization, and how to tune subsample size and ridge penalty efficiently for large datasets.
Approach
Treats subsampling as a weighted regression and uses asymptotic freeness with S-transforms to show weighted ridge estimators on pretrained features are asymptotically equivalent to unweighted ridge along paths matching effective degrees of freedom; for subsampling this gives (1-odf)(1-λ/μ)=(1-k/n), extended to linear, random, and kernel features and ensembles.
Key results
- Theory gives the subsampling path (1-odf)(1-λ/μ)=(1-k/n), so subsampled ridge needs smaller regularization than full-data ridge.
- Ensemble risk decomposes as R_M ≈ R_∞ + C/M, so ensemble risk error decays at rate 1/M.
- Optimal subsample size satisfies k* ≤ rank(G_I), and the proposed cross-validation (CGCV + extrapolation) estimates risk close to the optimal risk with subsampling ratio 0.01 on real pretrained features.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase the weighted-estimator ensemble size M | Weighted ridge ensembles · In-distribution and out-of-distribution prediction | prediction risk: decreases at a rate of 1/M | Helped |
| Tune subsample size using extrapolated cross-validation risk estimates | Subsampled pretrained representations · Real-world datasets | prediction risk estimate accuracy | Helped |
| Use a subsampling ratio k/n of 0.01 with a sufficiently large ensemble | Subsampled pretrained representations · Real-world datasets | prediction risk: 0.01 | No clear effect |
Surprising findings
- The optimal subsample size is never larger than the rank of the Gram matrix, so for n much larger than p, subsampled ensembles need no more than p observations per base estimator to reach optimal risk.
Limitations
- Results apply only to ridge regression, not general estimators or GLMs.
- Assumes asymptotic freeness between weights and feature matrices, which is hard to test directly.
- Equivalences hold asymptotically as n, p go to infinity.
Experimental setup
Datasets: CIFAR-100 Flowers-102
Benchmarks: CIFAR-100 Flowers-102
Metrics: prediction risk degrees of freedom estimator equivalence
Baselines: full-data ridge regression
Builds on: free sketching equivalence random features generalized cross-validation
Closest papers by meaning
- Mirror, Mirror of the Flow: How Does Regularization Shape Implicit Bias? (ICML 2025 · similarity 0.88)
- Estimating Implicit Regularization in Deep Learning (NeurIPS 2026 · similarity 0.88)
- Generalized equivalences between subsampling and ridge regularization (NeurIPS 2023 · similarity 0.86)
- Penalising the biases in norm regularisation enforces sparsity (NeurIPS 2023 · similarity 0.86)
- Decoupled Weight Decay Regularization (ICLR 2019 · similarity 0.86)
- Deep Weight Factorization: Sparse Learning Through the Lens of Artificial Symmetries (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.