Research map / Architectures and efficiency / Pruning and efficient tuning
Efficient Model Editing with Task-Localized Sparse Fine-tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TaLoS, a sparse fine-tuning method that updates only the least gradient-sensitive (Fisher-diagonal) parameters of a pre-trained model to build sparse task vectors that are weight-disentangled, without explicit linearization.
Problem
Task arithmetic needs task vectors that compose without interference, but existing approaches rely on costly network linearization and do not guarantee weight disentanglement (function localization) during fine-tuning.
Approach
Identify parameters with consistently low Fisher-information sensitivity across tasks in a pre-trained model (estimated per-task via iterative pruning-at-initialization-style mask calibration) and sparsely fine-tune only those, which promotes linear behavior and function localization without explicit linearization or cross-task data sharing.</!--core_idea>
Key results
- On task addition, TaLoS reaches 90.73% normalized accuracy on CLIP ViT-B/32 versus 86.08% for the next best (L-LoRA), and 97.22% on T5-Small.
- TaLoS achieves the lowest target accuracy in task negation on most vision models, e.g. 10.58% on ViT-B/16 and 10.68% on ViT-L/14.
- TaLoS has the lowest total iteration time (0.3263 s on ViT-L/14 vs 0.7588 s for L-LoRA) and peak memory (7.8 GiB vs 9.7 GiB for L-LoRA on ViT-L/14).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TaLoS task vectors rather than the strongest reported baseline | task addition with ViT-B/32, ViT-B/16, ViT-L/14, T5-Small, T5-Base, and T5-Large · Task Addition | normalized accuracy: 86.08 -> 90.73; 87.29 -> 91.41; 93.01 -> 95.20; 95.34 -> 97.22; 92.70 -> 95.87; 88.14 -> 90.61 | Helped |
| Use TaLoS task vectors rather than the strongest reported baseline | task negation with ViT-B/32, ViT-B/16, ViT-L/14, T5-Small, T5-Base, and T5-Large · Task Negation | target accuracy: 11.20 -> 11.03; 10.97 -> 10.58; 10.86 -> 10.68; 42.53 -> 39.64; 40.06 -> 38.49; 41.26 -> 37.20 | Helped |
| Use TaLoS sparse fine-tuning rather than the compared fine-tuning methods | ViT-B/32, ViT-L/14, and T5-Large · Fine-tuning | total iteration time: 0.3722 -> 0.1644; 0.5235 -> 0.3263; 0.7588 -> 0.4867 | Helped |
| Use TaLoS sparse fine-tuning rather than the compared fine-tuning methods | ViT-B/32, ViT-L/14, and T5-Large · Fine-tuning | peak memory usage: 5.3 -> 4.7; 9.7 -> 7.8; 18.2 -> 12.1 | Helped |
| Increase sparsity from 0% toward 90% (ablation) | TaLoS · Task arithmetic across vision and language experiments | task arithmetic performance | Helped |
| Increase sparsity beyond 90% (ablation) | TaLoS · Task arithmetic across vision and language experiments | task arithmetic performance | Hurt |
| Use 90% sparsity rather than 0% sparsity (ablation) | TaLoS fine-tuning · Task arithmetic across vision and language experiments | single-task accuracy | Hurt |
| Select least-sensitive parameters rather than task-specific parameters | TaLoS versus LoTA · Vision weight disentanglement | disentanglement error | Helped |
| Use TaLoS rather than LoTA | task vector fine-tuning · Task addition and task negation | single-task accuracy | Mixed |
Surprising findings
- Linearization alone does not guarantee function localization; linearized task functions can still activate on arbitrary inputs.
- On NLP tasks, all methods show natural function localization, so conclusions from vision alone may be misleading.
Limitations
- Sparsity ratio is a hyperparameter tuned per task (optimal 90-99%), and tuning requires task-level validation.
- Gradient-based mask calibration adds overhead and uses full-precision gradients, with no memory-optimized estimator implemented.
- Learning in the sparse subspace caps single-task accuracy slightly below full fine-tuning at lower model scales.
Experimental setup
Datasets: Cars DTD EuroSAT GTSRB MNIST RESISC45 SUN397 SVHN QASC WikiQA QuaRTz PAWS
Benchmarks: Task Arithmetic task addition (vision and NLP) Task Negation (ImageNet control for vision, RTE control for NLP)
Metrics: absolute accuracy normalized accuracy target accuracy control accuracy disentanglement error iteration time peak memory usage
Baselines: Non-linear FT Linearized FT TIES-Merging TALL Mask / Consensus DARE Breadcrumbs L-LoRA LoTA
Builds on: task arithmetic weight disentanglement linearized fine-tuning LoTA pruning-at-initialization
Closest papers by meaning
- Training-Free Task Vectors for LLM Behavioral Control (NeurIPS 2026 · similarity 0.89)
- When is Task Vector Provably Effective for Model Editing? A Generalization Analysis of Nonlinear Transformers (ICLR 2025 · similarity 0.88)
- Joint Localization and Activation Editing for Low-Resource Fine-Tuning (ICML 2025 · similarity 0.87)
- Resolving Lexical Bias in Model Editing (ICML 2025 · similarity 0.87)
- Parameter-Efficient Multi-Task Model Fusion with Partial Linearization (ICLR 2024 · similarity 0.86)
- CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuning (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.