Research map / Learning theory and algorithms / Kernels and regression theory
A New Neural Kernel Regime: The Inductive Bias of Multi-Task Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
placeholder
Problem
Single-task ReLU network weight-decay interpolation has non-unique solutions in a non-Hilbertian Banach space (BV2); it is unclear how multi-task training changes the learned function and its inductive bias.
Approach
With multiple tasks sharing the same inputs, the minimal-representational-cost interpolant for univariate inputs is almost always unique and equals connect-the-dots linear interpolation, which is the minimum-norm interpolant in H1 (a kernel method). For multivariate inputs with many tasks, the regularizer approximates a weighted l2 penalty, giving an approximate RKHS equivalence over the optimal n
Key results
- x
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train on multiple tasks rather than a single task | Univariate shallow ReLU network with weight decay · Univariate interpolation | uniqueness of the optimal interpolating function | Helped |
| Train on multiple tasks rather than a single task | Univariate shallow ReLU network with weight decay · Univariate interpolation | agreement with connect-the-dots interpolant | Helped |
| Increase the number of tasks from T = 1 to T = 101 | Multivariate shallow ReLU network · Two-dimensional synthetic interpolation | variation across learned solutions | Helped |
| Increase the number of tasks from T = 1 to T = 101 | Multivariate shallow ReLU network · Student-teacher synthetic data | recovery of the task-generating function | Hurt |
| Use large numbers of tasks, approximating the output-weight penalty by weighted ℓ2 regularization | Multivariate shallow ReLU network · Multivariate-input learning | approximation to an RKHS solution | Helped |
Surprising findings
- Multi-task training produces neuron sharing even when tasks are unrelated or random, with each neuron contributing to all outputs.
- The unique univariate multi-task solution is generally not the sparsest interpolant, unlike single-task solutions with the sparsity-favoring structure.
Limitations
- Univariate multi-task result is proven only for d=1; the multivariate kernel equivalence is approximate and holds asymptotically as the number of tasks grows.
- Kernel in the multivariate case is not precisely characterized.
- Results characterize global minimizers; convergence of gradient descent to them is not proven.
Experimental setup
Datasets: synthetic univariate interpolation data synthetic two-dimensional square dataset with random Bernoulli extra tasks synthetic student-teacher 5D ReLU data
Metrics: representational cost interpolation constraint satisfaction solution variability across initializations
Baselines: single-task ReLU network training minimum-norm interpolation in BV2 Banach space
Builds on: Parhi and Nowak representer theorem Savarese et al. 2019 ReLU weight decay characterization Shenouda et al. 2024 vector-valued neural network function spaces Hanin 2021/2022 univariate ReLU solutions De Boor and Lynch Sobolev kernel
Closest papers by meaning
- Provable Multi-Task Representation Learning by Two-Layer ReLU Neural Networks (ICML 2024 · similarity 0.86)
- Stable Minima of ReLU Neural Networks Suffer from the Curse of Dimensionality: The Neural Shattering Phenomenon (NeurIPS 2025 · similarity 0.85)
- Simplicity Bias and Optimization Threshold in Two-Layer ReLU Networks (ICML 2025 · similarity 0.85)
- Stable Minima Cannot Overfit in Univariate ReLU Networks: Generalization by Large Step Sizes (NeurIPS 2024 · similarity 0.85)
- Inductive biases of multi-task learning and finetuning: multiple regimes of feature reuse (NeurIPS 2024 · similarity 0.85)
- Global Minimizers of $\ell^p$-Regularized Objectives Yield the Sparsest ReLU Neural Networks (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.