Research map / Label-efficient and robust learning / Recognition and long-tail
Knowledge Composition using Task Vectors with Learned Anisotropic Scaling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes aTLAS, which learns per-parameter-block scalar coefficients over CLIP task vectors (anisotropic scaling) to compose knowledge for task arithmetic, few-shot recognition, test-time adaptation and PEFT.
Problem
Task arithmetic with task vectors uses a single isotropic scaling factor found by search, which ignores that different parameter blocks carry different knowledge; the paper asks how parameter blocks can be combined to enhance knowledge composition and transfer in low-data regimes.
Approach
Learn one scalar coefficient per parameter block for each task vector (block-diagonal anisotropic scaling), so that the learned linear combination of task vectors acts as a low-dimensional parameter-efficient adaptation; blockwise gradient selection allows mixing blocks from different task vectors.
Key results
- On task addition with ViT-B/32, standard task vectors with learned anisotropic scaling reach 84.98% absolute accuracy versus 70.12% for the searched isotropic baseline.
- Test-time adaptation with UFM and aTLAS reaches 66.9% vs 60.4% zero-shot on 22 datasets (about +6.5 points), with ~3.5k learnable parameters vs 30k for LayerNorm tuning.
- In 1-shot few-shot recognition across 22 datasets, aTLAS (standard) reaches 66.2% vs Tip-Adapter 64.3% and LP++ 64.5%; combined with LP++ it reaches 68.9-69.3%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| learn anisotropic rather than searched isotropic scaling for standard task-vector addition | CLIP task-vector composition · Task addition (8 datasets, ViT-B/32) | absolute accuracy: 70.12 -> 84.98 | Helped |
| learn anisotropic rather than searched isotropic scaling for standard task-vector negation | CLIP task-vector composition · Task negation (8 datasets, ViT-B/32) | target accuracy: 23.22 -> 18.76 | Helped |
| use aTLAS instead of LayerNorm tuning | CLIP ViT-B/32 test-time adaptation · SimCLR | accuracy: $60.4 \pm 0.0$ -> $62.7 \pm 0.1$ | Helped |
| use aTLAS instead of LayerNorm tuning | CLIP ViT-B/32 test-time adaptation · SAR | accuracy: $61.2 \pm 0.1$ -> $62.9 \pm 0.0$ | Helped |
| use aTLAS instead of LayerNorm tuning | CLIP ViT-B/32 test-time adaptation · UFM | accuracy: $62.2 \pm 0.1$ -> $\mathbf{66.9} \pm 0.1$ | Helped |
| combine aTLAS with LP++ | CLIP few-shot recognition · 22 datasets, 1-shot | accuracy: $ 66.2 \pm 0.2$ -> $ 68.9 \pm 0.2$ | Helped |
| use linearised instead of standard task vectors with aTLAS | CLIP few-shot recognition · 22 datasets, 1-shot | accuracy: $ 66.2 \pm 0.2$ -> $64.6 \pm 0.1$ | Hurt |
| use rank 4 LoRAs instead of standard task vectors (ablation) | aTLAS few-shot recognition · 22 datasets, 1-shot | accuracy: $66.0 \pm 0.2$ -> $64.4 \pm 0.1$ | Hurt |
| increase LoRA task-vector rank from 4 to 64 (ablation) | aTLAS few-shot recognition · 22 datasets, 1-shot | accuracy: $64.4 \pm 0.1$ -> $65.4 \pm 0.1$ | Helped |
| restrict learned coefficients to weight matrices rather than all parameter blocks (ablation) | aTLAS task negation · ViT-B/32, eight target datasets | target accuracy: 18.76 -> 20.14 | Hurt |
| add L1 regularisation to learned coefficients (ablation) | aTLAS task addition · 8 datasets, ViT-B/32 | absolute accuracy: $84.23$ vs. $84.98$ | Hurt |
| select parameter blocks across task vectors using blockwise gradients instead of selecting whole task vectors (ablation) | aTLAS few-shot recognition · 16-shot, low task-vector budgets | accuracy | Helped |
Surprising findings
- Learned coefficients on weight matrices alone achieve nearly the same task addition accuracy as all parameter blocks (84.17 vs 84.98 on ViT-B/32), suggesting weights carry most knowledge.
- Standard task vectors with learned anisotropic scaling outperform linearised ones in task addition, reversing the ranking seen with isotropic search.
Limitations
- Task vectors are defined relative to a specific pre-trained model, so composition does not transfer across architectures.
- Combining many task vectors consumes substantial GPU memory when training larger models.
- Learned coefficients reported without regularisation are harder to interpret; L1 regularisation is needed for clarity, causing small accuracy drops.
Experimental setup
Datasets: Stanford Cars DTD EuroSAT GTSRB MNIST RESISC45 SUN397 SVHN CIFAR10 CIFAR100 ImageNet STL10
Benchmarks: Eight-dataset task arithmetic (negation and addition) 22-dataset few-shot recognition ImageNet OOD (ImageNet-A, R, Sketch, V2) 22-dataset test-time adaptation
Metrics: accuracy relative accuracy disentanglement error accuracy improvement over pre-trained
Baselines: Task arithmetic search (isotropic) Linearised task vectors with search Tip-Adapter LP++ LayerNorm tuning LoRA Full-parameter fine-tuning SAR SimCLR
Builds on: task arithmetic linearised fine-tuning Tip-Adapter LP++ LoRA
Closest papers by meaning
- Variational Task Vector Composition (NeurIPS 2025 · similarity 0.88)
- Distributional Alignment as a Criterion for Designing Task Vectors in In-Context Learning (NeurIPS 2026 · similarity 0.87)
- Learning Task-Agnostic Representations through Multi-Teacher Distillation (NeurIPS 2025 · similarity 0.86)
- Task Vector Geometry Underlies Dual Modes of Task Inference in Transformers (NeurIPS 2026 · similarity 0.86)
- Amalgamating Multi-Task Models with Heterogeneous Architectures (AAAI 2024 · similarity 0.86)
- Task Groupings Regularization: Data-Free Meta-Learning with Heterogeneous Pre-trained Models (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.