Research map / Learning theory and algorithms / Kernels and regression theory

Transfer Learning of Linear Regression with Multiple Pretrained Models: Benefiting from More Pretrained Models via Overparameterization Debiasing

NeurIPS 2026 · Learning theory and algorithms › Kernels and regression theory · approach: High-dimensional asymptotic generalization · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Analyzes transfer learning of linear regression from multiple least-squares pretrained models via a penalized target fit, derives asymptotic test error via random matrix theory, and proposes an overparameterization debiasing scaling of the assumed task-relation operators.

Problem

Transfer learning to a target linear regression task from several pretrained least-squares source models, which may be overparameterized; overparameterized min-l2-norm pretrained models carry an attenuation bias that limits the benefits of using many pretrained models.

Approach

Extends single-model penalized transfer learning to multiple pretrained models with a summed distance penalty, and proposes debiasing overparameterized pretrained models by scaling the assumed task-relation operator by the inverse overparameterization level (n~/d), with a validation-tuned scalar when n~ is unknown.

Key results

• Transfer with multiple overparameterized pretrained models can require sufficiently many models to beat ridge regression; theory gives necessary condition sigma_eta^2 + sigma_xi^2 > m b implies no benefit.
• Overparameterization debiasing (scaling task relation by n~/d) makes transfer consistent as m grows: test error approaches Bayes error sigma_epsilon^2 for overparameterized sources.
• Debiasing is beneficial when (sigma_eta^2 + d sigma_xi^2/(d-n~-1))(d/n~ + 2d/(d-n~)) < (m-1-d/n~) b, requiring m > 1 + d/n~.

Surprising findings

• Overparameterized pretrained models make multi-model transfer inconsistent: the error stays above the Bayes error even as the number of pretrained models grows.
• Empirically chosen debiasing scalar can exceed 1 for underparameterized or few overparameterized sources, acting as implicit ridge regularization.

Limitations

• Analysis assumes isotropic Gaussian source inputs and Gaussian target parameter prior in the main theory; anisotropic case handled only via a lemma and empirical scalar tuning.
• Theory assumes a fixed finite number m of pretrained models in the asymptotic regime and known/orthonormal task relations for the consistency results.
• Debiasing with unknown n~ requires a validation set to tune the scaling factor.

Experimental setup

Datasets: synthetic Gaussian linear regression data

Benchmarks: synthetic linear regression (simulated)

Metrics: expected test squared error bias-variance decomposition

Baselines: single pretrained model transfer minimum l2-norm least squares optimally tuned ridge regression null estimator debiased vs non-debiased multi-model transfer

Builds on: single-pretrained-model linear transfer learning (Theorem 6.2 analysis) minimum l2-norm least squares ridge regression

Closest papers by meaning

• Transfer Learning for Benign Overfitting in High-Dimensional Linear Regression (NeurIPS 2025 · similarity 0.88)
• Transfer Learning Meets Functional Linear Regression: No Negative Transfer Under Posterior Drift (AAAI 2025 · similarity 0.88)
• Minimum-Norm Interpolation Under Covariate Shift (ICML 2024 · similarity 0.87)
• Universality in Transfer Learning for Linear Models (NeurIPS 2024 · similarity 0.86)
• Wasserstein Transfer Learning (NeurIPS 2025 · similarity 0.86)
• A High-Dimensional Statistical Method for Optimizing Transfer Quantities in Multi-Source Transfer Learning (NeurIPS 2025 · similarity 0.86)