Research map / Architectures and efficiency / Pruning and efficient tuning
A Second-Order Perspective on Model Compositionality and Incremental Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Gives a second-order Taylor analysis of when fine-tuned models can be composed by weight averaging, and derives two incremental algorithms (ITA and IEL) that regularize with a diagonal-Fisher term to stay in the pre-training basin.
Problem
Identifying conditions under which individually fine-tuned task vectors can be composed (averaged) into an accurate multi-task model in standard non-linear networks, and how to learn such composable modules incrementally in class-incremental settings.
Approach
Uses a second-order Taylor approximation of the loss around pre-training weights to show the composed model's loss is bounded by the average of individual losses and derives an exact gap term Omega; this motivates a Fisher-weighted (EWC-like) regularizer for individual training (ITA) and an ensemble-level regularizer for joint training of the composition (IEL).
Key results
- ITA-LoRA achieves 86.60% final accuracy on Split MIT-67 and 77.79% on Split ImageNet-R, outperforming baselines such as InfLoRA (76.97% on IN-R) and SEED.
- IEL-FFT reaches 80.09% on Split ImageNet-R, the best result on that dataset in the comparison.
- Removing the proposed regularizer drops ITA-FFT from 76.43% to 8.61% on Split ImageNet-R, showing the regularization is essential.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the proposed regularization to ITA-FFT instead of removing it (ablation) | ITA-FFT · Split ImageNet-R, Split CIFAR-100, Split CUB-200, Split Caltech-256, Split MIT-67, Split RESISC45, Split CropDiseases | Final Accuracy: 8.61 -> 76.43; 17.59 -> 89.38; 10.47 -> 84.80; 12.76 -> 92.32; 12.01 -> 85.35; 17.17 -> 80.50; 20.64 -> 91.81 | Helped |
| Add the proposed regularization to ITA-LoRA instead of removing it (ablation) | ITA-LoRA · Split ImageNet-R, Split CIFAR-100, Split CUB-200, Split Caltech-256, Split MIT-67, Split RESISC45, Split CropDiseases | Final Accuracy: 50.17 -> 77.79; 66.58 -> 89.96; 60.58 -> 85.55; 74.87 -> 92.65; 52.74 -> 86.60; 37.59 -> 82.00; 55.86 -> 95.85 | Helped |
| Apply the proposed regularization only to the CLS classification layer rather than all layers (ablation) | ITA-FFT · Split ImageNet-R, Split CIFAR-100, Split CUB-200, Split Caltech-256, Split MIT-67, Split RESISC45, Split CropDiseases | Final Accuracy: 76.43 -> 76.00; 89.38 -> 87.60; 84.80 -> 83.54; 92.32 -> 91.04; 85.35 -> 81.45; 80.50 -> 75.26; 91.81 -> 77.00 | Hurt |
| Apply the proposed regularization only to the CLS classification layer rather than all layers (ablation) | ITA-LoRA · Split ImageNet-R, Split CIFAR-100, Split CUB-200, Split Caltech-256, Split MIT-67, Split RESISC45, Split CropDiseases | Final Accuracy: 77.79 -> 77.33; 89.96 -> 90.03; 85.55 -> 85.55; 92.65 -> 92.59; 86.60 -> 84.86; 82.00 -> 80.64; 95.85 -> 96.22 | Mixed |
| Use ITA-LoRA instead of InfLoRA | ViT-B/16 · Split ImageNet-R | Final Accuracy: 76.97 -> 77.79 | Helped |
| Use ITA-IA instead of L2P | ViT-B/16 · Split CIFAR-100 | Final Accuracy: 87.32 -> 90.66 | Helped |
| Use ITA-IA instead of SEED | ViT-B/16 · Split CUB-200 | Final Accuracy: 85.35 -> 85.67 | Helped |
| Use IEL-IA instead of L2P | ViT-B/16 · Split Caltech-256 | Final Accuracy: 91.22 -> 92.70 | Helped |
| Use ITA-LoRA instead of SEED | ViT-B/16 · Split MIT-67 | Final Accuracy: 86.34 -> 86.60 | Helped |
| Use ITA-IA instead of InfLoRA | ViT-B/16 · Split RESISC45 | Final Accuracy: 79.92 -> 83.73 | Helped |
| Use ITA-LoRA instead of DER++ | ViT-B/16 · Split CropDiseases | Final Accuracy: 98.77 -> 95.85 | Hurt |
Surprising findings
- Full fine-tuning collapses without the regularizer (8.61% on IN-R), while PEFT methods such as LoRA retain decent accuracy (50.17%).
- The IEL composed ensemble suffers a severe accuracy drop when members are removed for specialization, unlike ITA.
Limitations
- Second-order approximation is anchored at pre-training weights and may become inaccurate as parameters drift.
- Relies on Gaussian weight-distribution and diagonal Fisher approximations.
- Focuses on closed-set classification models rather than self-supervised or open-vocabulary models like CLIP.
Experimental setup
Datasets: ImageNet-R CIFAR-100 CUB-200 Caltech-256 MIT-67 RESISC45 CropDiseases ImageNet21K
Benchmarks: Split ImageNet-R Split CIFAR-100 Split CUB-200 Split Caltech-256 Split MIT-67 Split RESISC45 Split CropDiseases
Metrics: final accuracy final forgetting zero-shot specialization accuracy zero-shot unlearning accuracy
Baselines: EWC LwF-MC DER++ L2P CODA SEED InfLoRA APT TMC
Builds on: task arithmetic model soups Elastic Weight Consolidation LoRA tangent model composition
Closest papers by meaning
- Separating the 'what' and 'how' of compositional computation to enable reuse and continual learning (NeurIPS 2025 · similarity 0.87)
- Scaling can lead to compositional generalization (NeurIPS 2025 · similarity 0.87)
- When does compositional structure yield compositional generalization? A kernel theory. (ICLR 2025 · similarity 0.86)
- Not Just Object, But State: Compositional Incremental Learning without Forgetting (NeurIPS 2024 · similarity 0.86)
- Position: A Theory of Deep Learning Must Include Compositional Sparsity (ICML 2025 · similarity 0.86)
- Compositional Few-Shot Class-Incremental Learning (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.