Research map / Architectures and efficiency / Pruning and efficient tuning
Understanding and Enforcing Weight Disentanglement in Task Arithmetic
Shangge Liu, Yuehan Yin, Lei Wang, Qi Fan, Yinghuan Shi, Wenbin Li, Yang Gao, Dacheng Tao
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add OrthoReg orthogonal regularization on weight updates | Non-linear Finetuning (ViT-L-14) · Task addition (8 tasks) | absolute accuracy: 84.07 -> 88.23 | Improved |
| add OrthoReg | Tangent Task Arithmetic (ViT-L-14) · Task addition (8 tasks) | absolute accuracy: 86.19 -> 87.52 | Improved |
| add OrthoReg | Attention-Only Fine-tuning (ViT-B-16) · Task addition (8 tasks) | absolute accuracy: 80.71 -> 83.37 | Improved |
| add OrthoReg | LoRA-ATT (ViT-B-16) · Task addition (8 tasks) | absolute accuracy: 75.51 -> 79.67 | Improved |
| add OrthoReg | Non-linear Finetuning (ViT-B-16) · Task addition (8 tasks) | normalized accuracy: 75.39 -> 93.62 | Improved |
| add OrthoReg | Attention-Only Fine-tuning (ViT-L-14) · Task addition (8 tasks) | normalized accuracy: 93.59 -> 100.05 | Improved |
| add OrthoReg | Tangent Task Arithmetic (ViT-B-16) · Task addition (8 tasks) | normalized accuracy: 87.48 -> 88.02 | No clear effect |
| add OrthoReg | Attention-Only Fine-tuning (ViT-L-14) · Task negation (8 tasks) | target accuracy: 24.85 -> 14.67 | Improved |
| add OrthoReg | Tangent Task Arithmetic (ViT-L-14) · Task negation (8 tasks) | target accuracy: 8.36 -> 8.36 | No clear effect |
| add OrthoReg | Tangent Task Arithmetic (ViT-B-32) · Task negation (8 tasks) | target accuracy: 11.47 -> 11.39 | No clear effect |
| add OrthoReg | Attention-Only Fine-tuning (ViT-L-14) · Task negation (8 tasks) | ImageNet control accuracy: 76.42 -> 75.40 | Mixed |
| add OrthoReg | Non-linear Finetuning (ViT-B-16) · Cosine similarity of task vectors | off-diagonal cosine similarity | Improved |
Closest papers by meaning
- Fine-Tuning Attention Modules Only: Enhancing Weight Disentanglement in Task Arithmetic (ICLR 2025 · similarity 0.89)
- Platonic Task Arithmetic (NeurIPS 2026 · similarity 0.86)
- Task Singular Vectors: Reducing Task Interference in Model Merging (CVPR 2025 · similarity 0.86)
- Disentangling Representations through Multi-task Learning (ICLR 2025 · similarity 0.86)
- Knowledge Composition using Task Vectors with Learned Anisotropic Scaling (NeurIPS 2024 · similarity 0.86)
- Task Vector Geometry Underlies Dual Modes of Task Inference in Transformers (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
