Research map / Architectures and efficiency / Pruning and efficient tuning
Distilling Linearized Behavior into Non-linear Fine-Tuning for Effective Task Arithmetic
Thomas Sommariva, Francesca Morandi, Simone Calderara, Angelo Porrello
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| distill linearized teacher activations into non-linear student with curvature regularization (DELTA) vs plain non-linear FT | CLIP ViT-B/32 non-linear fine-tuning · 8-Vision task addition (alpha=1.0) | absolute accuracy: 32.0 -> 88.3 | Improved |
| add linearized teacher distillation on top of curvature regularization (ablation) | DELTA · 8-Vision task addition, ViT-B/32 | mean absolute accuracy: 81.1 -> 88.3 | Improved |
| add EK-FAC curvature regularization on top of distillation (ablation) | DELTA · 8-Vision task addition, ViT-B/32 | mean absolute accuracy: 84.2 -> 88.3 | Improved |
| add distillation and curvature regularization on T5-Base (ablation) | DELTA · 6-NLI task addition, T5-Base | mean absolute accuracy: 78.9 -> 82.4 (w/o teacher vs full); 78.5 -> 82.4 (w/o EK-FAC vs full) | Improved |
| add linearized teacher distillation for task negation (ablation) | DELTA · 8-Vision task negation, ViT-B/32 | target accuracy: 14.2 -> 9.6 | Improved |
| add curvature regularization for task negation (ablation) | DELTA · 8-Vision task negation, ViT-B/32 | target accuracy: 16.8 -> 9.6 | Improved |
| sample teacher along path (alpha sampling) instead of fixing alpha=1 (ablation) | DELTA distillation · 8-Vision, linearization error and alpha sweep | linearization error and robustness to alpha | Improved |
| use ImageNet-1k instead of ImageNet-21k as reference dataset for curvature (ablation) | DELTA · 8-Vision task addition, ViT-B/32 (alpha=1.0) | absolute accuracy: 88.3 -> 87.5 | Hurt |
| use PAWS instead of C4 as reference dataset (ablation) | DELTA · 6-NLI task addition, T5-Base (alpha=1.0) | absolute accuracy: 82.3 -> 80.3 | Hurt |
| use QQP instead of C4 as reference dataset (ablation) | DELTA · 6-NLI task addition, T5-Base (alpha=1.0) | absolute accuracy: 82.3 -> 81.6 | Hurt |
| distill linearized teacher into a LoRA student instead of post-hoc merging (Iso-C, TSV-M, Core) | LoRA task addition · 8-Vision, ViT-B/32 | absolute accuracy: 77.9 -> 87.5 | Improved |
| DELTA versus linear-regime methods (tau-Jp, TAK) for task negation | task negation · 8-Vision task negation, ViT-B/32 | target accuracy: 2.8 -> 9.6 (TAK vs DELTA) | Hurt |
Closest papers by meaning
- Parameter-Efficient Multi-Task Model Fusion with Partial Linearization (ICLR 2024 · similarity 0.90)
- Variational Task Vector Composition (NeurIPS 2025 · similarity 0.89)
- Fine-Tuning Attention Modules Only: Enhancing Weight Disentanglement in Task Arithmetic (ICLR 2025 · similarity 0.89)
- Leveraging Submodule Linearity Enhances Task Arithmetic Performance in LLMs (ICLR 2025 · similarity 0.89)
- Efficient Model Editing with Task-Localized Sparse Fine-tuning (ICLR 2025 · similarity 0.89)
- Training-Free Task Vectors for LLM Behavioral Control (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
