Research map / Label-efficient and robust learning / Recognition and long-tail
Inductive biases of multi-task learning and finetuning: multiple regimes of feature reuse
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives implicit/explicit regularization penalties for multi-task learning and pretraining+finetuning in diagonal linear and single-hidden-layer ReLU networks, identifies a "nested feature selection" regime for finetuning, and shows weight rescaling after pretraining improves ResNet-18 finetuning on CIFAR-100.
Problem
Inductive biases arising from training neural networks on multiple tasks, either simultaneously (MTL) or sequentially (pretraining then finetuning, PT+FT), are poorly characterized theoretically.
Approach
Derives closed-form regularization penalties for MTL (a group-lasso-like l1,2 norm) and PT+FT (a penalty depending on pretrained feature magnitudes, with an l-order plus feature-dependence conservation law), revealing a nested feature selection regime in which finetuning sparsely reuses pretrained features; weight rescaling after pretraining is proposed to elicit this regime.
Key results
- Proves l-order + feature dependence = 1 for MTL and PT+FT penalties in both diagonal linear and ReLU networks (the conservation law).
- In teacher-student ReLU experiments, PT+FT outperforms STL and MTL when main-task features are correlated with auxiliary features, whereas MTL only helps in the low-sample regime.
- In ResNet-18 on CIFAR-100, rescaling weights by values less than 1 after pretraining improves finetuning accuracy, and rescaled models show reduced participation ratio and ENSD signatures of nested feature selection.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PT+FT instead of STL when auxiliary and main task features are fully shared | Teacher-student models · Diagonal linear and ReLU tasks with fully overlapping features | generalization loss | Helped |
| Use MTL instead of STL when auxiliary and main task features are fully shared | Teacher-student models · Diagonal linear and ReLU tasks with fully overlapping features | generalization loss | Helped |
| Use PT+FT instead of STL on tasks with partially overlapping features | Teacher-student models · Diagonal linear and ReLU tasks with partially overlapping features | generalization loss | Helped |
| Use MTL instead of STL on tasks with partially overlapping features | Teacher-student models · Diagonal linear and ReLU tasks with partially overlapping features | generalization loss | Helped |
| Rescale pretrained ResNet-18 weights by a factor less than 1 before finetuning | ResNet-18 · CIFAR-100 two-category primary task | accuracy | Helped |
| Rescale pretrained ViT weights before finetuning | Vision Transformer · CIFAR-100 two-category primary task | generalization performance | Hurt |
| Use PT+FT instead of STL | Vision Transformer · CIFAR-100 two-category primary task | performance | Helped |
| Use MTL instead of STL | Vision Transformer · CIFAR-100 two-category primary task | performance | No clear effect |
| Finetune with PT+FT rather than MTL on correlated, distinct task features | ReLU networks · Teacher-student tasks with correlated but distinct features | generalization loss | Mixed |
| Rescale pretrained ReLU-network weights by 10.0 before finetuning (ablation) | PT+FT ReLU networks · Nested feature selection task with one main-task feature | sparsity of learned solution | Hurt |
| Rescale pretrained ReLU-network weights by 0.1 before finetuning (ablation) | PT+FT ReLU networks · Task where main-task features are a subset of auxiliary-task features | feature alignment score | Hurt |
Surprising findings
- Vision Transformers already show signatures of nested feature selection without rescaling and do not benefit from weight rescaling, unlike ResNets.
- PT+FT in ReLU networks benefits from correlated (not just identical) auxiliary features, while MTL does not.
Limitations
- Theory is for shallow diagonal linear and single-hidden-layer ReLU networks with explicit regularization and idealized assumptions; deep-network phenomena are only empirically checked.
- Optimal rescaling value differs between ResNets and ViTs and is not predicted by the theory.
- Only MSE-based linear and ReLU settings are analyzed; cross-entropy and different task objectives are left to future work.
Experimental setup
Datasets: CIFAR-100 ImageNet synthetic teacher-student data
Benchmarks: CIFAR-100 ImageNet
Metrics: generalization loss accuracy participation ratio effective number of shared dimensions (ENSD) l-order feature dependence inertia
Baselines: single-task learning linear probing NTK MTL PT+FT
Builds on: diagonal linear network implicit bias (Woodworth et al.) group lasso F1 norm characterization of shallow ReLU networks
Closest papers by meaning
- Provable Multi-Task Representation Learning by Two-Layer ReLU Neural Networks (ICML 2024 · similarity 0.88)
- On the Emergence of Cross-Task Linearity in Pretraining-Finetuning Paradigm (ICML 2024 · similarity 0.87)
- Pay Attention to Small Weights (NeurIPS 2025 · similarity 0.86)
- Simplicity Bias and Optimization Threshold in Two-Layer ReLU Networks (ICML 2025 · similarity 0.86)
- Feature Averaging: An Implicit Bias of Gradient Descent Leading to Non-Robustness in Neural Networks (ICLR 2025 · similarity 0.86)
- Do We Always Need the Simplicity Bias? Looking for Optimal Inductive Biases in the Wild (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.