Research map / Architectures and efficiency / RNNs and normalization
On the Emergence of Cross-Task Linearity in Pretraining-Finetuning Paradigm
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Identifies Cross-Task Linearity, where weight interpolation of models finetuned from a shared pretrained checkpoint on different tasks yields approximately linear feature interpolation, and uses it to explain model averaging and task arithmetic while attributing the effect to pretraining.
Problem
Understanding why models finetuned from a common pretrained checkpoint on different tasks remain linearly connected in feature space, and what this implies for model merging/editing such as model averaging and task arithmetic.
Approach
Extends LMC and LLFC from same-task spawned models to different-task finetuned models from a shared pretrained checkpoint, showing features of weight-interpolated models approximately equal linear interpolations of per-model features (CTL), not just proportional; interprets model averaging and task arithmetic as feature-space operations and links CTL's emergence to pretraining and loss-landscape f
Key results
- Across ViT-B/32 and ResNet-18 settings, E[1-cosine] between weight-interpolated features and interpolated features approaches 0 across layers and alpha, while the baseline cosine between the two finetuned models deviates from 0.
- Increasing the number of pretraining epochs and the fraction of pretraining-like samples in finetuning both reduce 1-cosine and push the coef toward 1, indicating pretraining promotes CTL.
- The stitched model that replaces pretrained features with the average of two finetuned models' features achieves comparable accuracy on both tasks, while single-task feature replacement only solves one task.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase pretraining epochs (ablation) | ResNet-18 on Split CIFAR-100 · Split CIFAR-100 | CTL feature-interpolation error | Helped |
| vary the number of finetuning epochs (ablation) | ResNet-18 on Split CIFAR-100 · Split CIFAR-100 | CTL feature-interpolation error | No clear effect |
| increase similarity between pretraining and finetuning tasks (ablation) | ResNet-18 on Split ImageNet · Split ImageNet | CTL feature-interpolation error | Helped |
| increase similarity between the two finetuning tasks (ablation) | ResNet-18 on Split ImageNet · Split ImageNet | CTL feature-interpolation error | No clear effect |
| average features from two task-specific models | task-vector addition · paired downstream tasks | accuracy | Helped |
| replace averaged task-specific features with features from either single task model (ablation) | task-vector addition · paired downstream tasks | accuracy | Hurt |
| negate a task vector | pretrained model with task arithmetic · the corresponding downstream task | accuracy | Hurt |
| negate a task vector | pretrained model with task arithmetic · pretraining task | accuracy | No clear effect |
Surprising findings
- LMC does not hold for models finetuned on different tasks even on the pretraining task, while LLFC/CTL still holds in feature space.
- Finetuning epochs have little effect on the emergence of CTL, and similarity between finetuning tasks has little effect, whereas pretraining-finetuning similarity matters.
Limitations
- Primarily empirical; the theoretical result is only a first step and does not fully prove CTL.
- Experiments focus on vision models (ResNet-18, ViT) and T5 checkpoints; CTL on large language models is left to future work.
- CTL is not claimed to be universal for arbitrary networks.
Experimental setup
Datasets: Rotated MNIST Split CIFAR-100 Split ImageNet CIFAR-10 ImageNet
Benchmarks: CIFAR-10 ImageNet Split CIFAR-100 Split ImageNet Rotated MNIST
Metrics: cosine similarity coef (projection length ratio) accuracy
Baselines: LMC LLFC logits ensemble
Builds on: Linear Mode Connectivity Layerwise Linear Feature Connectivity task arithmetic model stitching
Closest papers by meaning
- Inductive biases of multi-task learning and finetuning: multiple regimes of feature reuse (NeurIPS 2024 · similarity 0.87)
- Mitigating Parameter Interference in Model Merging via Sharpness-Aware Fine-Tuning (ICLR 2025 · similarity 0.87)
- Pay Attention to Small Weights (NeurIPS 2025 · similarity 0.86)
- Features are fate: a theory of transfer learning in high-dimensional regression (ICML 2025 · similarity 0.86)
- The Weight Gram Matrix Captures Sequential Feature Linearization in Deep Networks (NeurIPS 2026 · similarity 0.86)
- Provable Multi-Task Representation Learning by Two-Layer ReLU Neural Networks (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.