Research map / Learning theory and algorithms / Kernels and regression theory

Transfer Learning for Benign Overfitting in High-Dimensional Linear Regression

NeurIPS 2025 · Learning theory and algorithms › Kernels and regression theory · approach: Double descent and overfitting · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a two-step Transfer Minimum-l2-Norm Interpolator (TM) that pre-trains a source-only MNI and fine-tunes on target data by minimal-distance interpolation, plus a CV-based ensemble (WTM) over informative sources, with non-asymptotic risk analysis and free-lunch covariate shift results.

Problem

Transfer learning for overparameterized interpolators: whether leveraging source tasks can further improve the out-of-sample generalization of minimum-l2-norm interpolators (benign overfitting) in high-dimensional linear regression under model and covariate shift.

Approach

Pre-train a source MNI and then fine-tune by taking the minimum-distance interpolant of the target data around it, so target-learned signal is retained in the target row space while source knowledge is transferred only into its null space; sources are selected by cross-validation and combined with CV-based weights.

Key results

• Transfer MNI outperforms target-only MNI under model shift when SSR<1 and SNR(1-SSR) exceeds p/(p-(n_q+1)); optimal transfer size n_q* = p-1-sqrt(p(p-1)/(SNR(1-SSR))) is given in closed form.
• Under free-lunch covariate shift (scaling source covariance by alpha>1), the variance inflation is reduced by a factor alpha while bias is unchanged or of the same order.
• In simulations with 50 replications, the CV-weighted ensemble WTM consistently outperforms individual TM estimates, pooled-MNI and SGD-based transfer, with pooled-MNI degrading catastrophically under distribution shift.

Measured effects reported in the paper

Surprising findings

• Pooled-MNI, although strong without distribution shift, suffers catastrophically under the model and covariate shifts considered.
• A source with negative transfer under covariate shift (TM for source 3 in one setting) is filtered out by the CV-based ensemble.

Limitations

• Consistency of the cross-validation-based informative source detection is not proved (left as future work).
• Covariate-shift bounds carry a reciprocal eigenvalue factor from the lack of simultaneous diagonalizability, which may loosen them.
• Analysis is limited to linear models; the RKHS extension is only sketched.

Experimental setup

Datasets: synthetic Gaussian linear regression data

Benchmarks: synthetic linear regression simulations

Metrics: excess risk bias variance

Baselines: target-only MNI pooled-MNI SGD_q pre-trained fine-tuning

Builds on: minimum-l2-norm interpolator benign overfitting theory (Bartlett et al.) pooled-MNI SGD-based transfer (pre-train then fine-tune) Dar et al. model shift cosine similarity analysis

Closest papers by meaning

• Minimum-Norm Interpolation Under Covariate Shift (ICML 2024 · similarity 0.94)
• Beyond Benign Overfitting in Nadaraya-Watson Interpolators (NeurIPS 2025 · similarity 0.89)
• Transfer Learning of Linear Regression with Multiple Pretrained Models: Benefiting from More Pretrained Models via Overparameterization Debiasing (NeurIPS 2026 · similarity 0.88)
• Rethinking Benign Overfitting in Two-Layer Neural Networks (ICML 2025 · similarity 0.87)
• On Hypothesis Transfer Learning of Functional Linear Models (ICML 2024 · similarity 0.87)
• Wasserstein Transfer Learning (NeurIPS 2025 · similarity 0.87)