Research map / Architectures and efficiency / Pruning and efficient tuning
Update Your Transformer to the Latest Release: Re-Basin of Task Vectors
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TransFusion, a data-free two-level weight-permutation (inter-head then intra-head spectral/LAP matching plus residual handling) that re-bases a Transformer task vector onto a new pretrained release so fine-tuning transfers without retraining.
Problem
Fine-tuned models become obsolete when the underlying pretrained foundation model is replaced by a new release; transferring the task vector to the new checkpoint without data or retraining is hard because of multi-head attention and residual connections in Transformers.
Approach
Extends Git Re-Basin-style weight matching to Transformers by matching attention heads via a permutation-invariant singular-value distance (inter-head LAP), then intra-head LAP permutations, and enforcing identical permutations on residual branches; the resulting permutation is applied to the task vector and added to the new base model.
Key results
- On visual tasks, TransFusion gives task-accuracy gains over zero-shot DataComp CLIP of +4.95 (EuroSAT), +1.10 (GTSRB), +3.64 (SVHN) and +0.21 (DTD), while Git Re-Basin and OT yield smaller or negative gains.
- On GLUE with a transported text encoder, TransFusion improves QQP by +6.50, SST2 by +5.96, RTE by +3.61 and CoLA by +2.49 over the zero-shot DataComp baseline, outperforming Git Re-Basin and OT.
- Loss along the linear interpolation path between the permuted CIFAR-10 ViT and the reference model stays low as alpha approaches 0, unlike OT and Git Re-Basin, supporting functional equivalence.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Transport the task vector with TransFusion permutations rather than Git Re-Basin | CLIP ViT-B/16 · EuroSAT | task accuracy change: +4.95 vs +0.95 | Helped |
| Transport the task vector with TransFusion permutations rather than Git Re-Basin | CLIP ViT-B/16 · DTD | task accuracy change: +0.21 vs -0.91 | Helped |
| Transport the task vector with TransFusion permutations rather than Git Re-Basin | CLIP ViT-B/16 · GTSRB | task accuracy change: +1.10 vs +0.76 | Helped |
| Transport the task vector with TransFusion permutations rather than Git Re-Basin | CLIP ViT-B/16 · SVHN | task accuracy change: +3.64 vs +0.79 | Helped |
| Transport the task vector with TransFusion permutations rather than Git Re-Basin | ViT-B-16 text encoder · QQP | accuracy change: +6.50 vs +3.58 | Helped |
| Transport the task vector with TransFusion permutations rather than Git Re-Basin | ViT-B-16 text encoder · SST2 | accuracy change: +5.96 vs +5.73 | Helped |
| Transport the task vector with TransFusion permutations rather than Git Re-Basin | ViT-B-16 text encoder · RTE | accuracy change: +3.61 vs +2.17 | Helped |
| Transport the task vector with TransFusion permutations rather than Git Re-Basin | ViT-B-16 text encoder · CoLA | accuracy change: +2.49 vs +1.44 | Helped |
| Use full two-stage attention alignment instead of brute-force head alignment (ablation) | TransFusion · EuroSAT | task accuracy change: +1.32 -> +4.95 | Helped |
| Use full two-stage attention alignment instead of brute-force head alignment (ablation) | TransFusion · GTSRB | task accuracy change: +0.60 -> +1.10 | Helped |
| Use full two-stage attention alignment instead of brute-force head alignment (ablation) | TransFusion · SVHN | task accuracy change: +3.39 -> +3.64 | Helped |
| Use full two-stage attention alignment instead of no attention alignment (ablation) | TransFusion · EuroSAT | task accuracy change: +2.22 -> +4.95 | Helped |
Surprising findings
- Brute-force inter-head matching performed worse than keeping the original head ordering (no attention alignment) on two of three visual tasks.
- Naive transport of the task vector without permutation (theta_B + tau) substantially degrades performance on several tasks, e.g. -22.00 on SVHN.
Limitations
- Performance on DTD is limited because the source task vector is weak (about 75% accuracy with few examples per class).
- Permutation matching relies on simple dot-product similarity for linear layers rather than activation-based matching.
- Only equal-architecture models are handled; mismatched layer counts are left to future work.
Experimental setup
Datasets: DTD EuroSAT GTSRB SVHN ImageNet-R GLUE CIFAR-10 CommonPool DataComp
Benchmarks: EuroSAT DTD GTSRB SVHN ImageNet-R QQP SST2 RTE CoLA
Metrics: task accuracy support accuracy loss along linear interpolation path
Baselines: Git Re-Basin Optimal Transport re-basin vanilla task vector transfer
Builds on: Git Re-Basin task arithmetic optimal transport re-basin
Closest papers by meaning
- Transferring Knowledge From Large Foundation Models to Small Downstream Models (ICML 2024 · similarity 0.85)
- Towards Few-Shot Adaptation of Foundation Models via Multitask Finetuning (ICLR 2024 · similarity 0.85)
- WeGeFT: Weight‑Generative Fine‑Tuning for Multi‑Faceted Efficient Adaptation of Large Models (ICML 2025 · similarity 0.85)
- Parameter Efficient Fine-tuning via Explained Variance Adaptation (NeurIPS 2025 · similarity 0.85)
- Low-Rank Rescaled Vision Transformer Fine-Tuning: A Residual Design Approach (CVPR 2024 · similarity 0.85)
- Foundation Models Secretly Understand Neural Network Weights: Enhancing Hypernetwork Architectures with Foundation Models (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.