Research map / Architectures and efficiency / Pruning and efficient tuning
RECAST: Reparameterized, Compact weight Adaptation for Sequential Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RECAST, which reparameterizes pretrained layer weights as combinations of shared frozen template banks with tiny per-module coefficients (<50 task params), reconstructed from pretrained weights via Neural Mimicry, for task-incremental learning.
Problem
Task-incremental learning adapters still require tens or hundreds of thousands of task-specific parameters, which is costly for resource-constrained edge devices and communication-limited settings.
Approach
Decompose each pretrained layer's weights into shared template banks and very few module-specific scaling coefficients, and learn this decomposition by reconstructing existing pretrained weights (Neural Mimicry) instead of training from scratch; per task only coefficients are tuned while templates stay frozen.
Key results
- RECAST on ViT-Small reaches 85.0% average TIL accuracy vs 82.4% baseline with about 2e-6 P task parameters; RECAST + AdaptFormer reaches 90.1%, best among compared methods.
- On ResNet-34, RECAST + Piggyback reaches 83.9% vs 82.9% for Piggyback alone; RECAST + CLR reaches 80.9% vs 79.9% for CLR alone.
- Neural Mimicry reconstructions of pretrained ViT-Small, ViT-Base, ViT-Large and DINO-Small reach 99-100% reconstruction similarity and near-identical ImageNet top-1 accuracy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use RECAST instead of the unadapted ResNet-34 baseline | ResNet-34 · Six-dataset average | Acc@top1 (%): 65.2 -> 66.3 ± 0.05 | Helped |
| add RECAST to Piggyback | Piggyback on ResNet-34 · Six-dataset average | Acc@top1 (%): 82.9 -> 83.9 ± 0.1 | Helped |
| add RECAST to CLR | CLR on ResNet-34 · Six-dataset average | Acc@top1 (%): 79.9 -> 80.9 ± 0.3 | Helped |
| use RECAST instead of the unadapted ViT-Small baseline | ViT-Small · Six-dataset average | Acc@top1 (%): 82.4 -> 85.0 ± 0.1 | Helped |
| add RECAST to AdaptFormer | AdaptFormer on ViT-Small · Six-dataset average | Acc@top1 (%): 89.0 -> 90.1 ± 0.2 | Helped |
| add RECAST to MeLo | MeLo on ViT-Small · Six-dataset average | Acc@top1 (%): 88.7 -> 89.6 ± 0.2 | Helped |
| add RECAST to RoSA Full - Att. | RoSA Full - Att. on ViT-Small · Six-dataset average | Acc@top1 (%): 88.6 -> 88.9 ± 0.08 | Helped |
| add RECAST to RoSA SpA MLP | RoSA SpA MLP on ViT-Small · Six-dataset average | Acc@top1 (%): 84.5 -> 85.4 ± 0.05 | Helped |
| use L2P instead of the unadapted ViT-Small baseline | ViT-Small · Six-dataset average | Acc@top1 (%): 82.4 -> 35.9 | Hurt |
| use VAE + MSE for weight reconstruction rather than the standard reconstruction (ablation) | RECAST ViT-Small · ImageNet-1K | Acc@top1: 74.6 -> 73.8 | Hurt |
| use VAE + SmoothL1 for weight reconstruction rather than the standard reconstruction (ablation) | RECAST ViT-Small · ImageNet-1K | Acc@top1: 74.6 -> 73.0 | Hurt |
| use SmoothL1 rather than MSE or KL divergence for reconstruction (ablation) | Neural Mimicry · Weight reconstruction | classification performance | Helped |
Surprising findings
- SmoothL1-trained reconstructions outperform VAE-based reconstructions on TIL accuracy, which the authors link to lower template diversity in VAE models.
Limitations
- Evaluated only on image classification in the task-incremental setting with six datasets.
- Reconstruction is bounded by the existing pretrained weights, and the paper does not explore novel parameter generation.
- Attention layers are harder to reparameterize than MLP layers, limiting gains there.
Experimental setup
Datasets: Oxford Flowers FGVC Aircraft MIT Scenes CIFAR100 CIFAR10 CUB-200 birds ImageNet-1K
Benchmarks: Flowers FGVC Aircraft MIT Scenes CIFAR100 CIFAR10 CUB-200 ImageNet-1K
Metrics: Acc@top1 task parameters per task reconstruction similarity
Baselines: EWC LwF GDumb ER-ACE DER++ HAT PiggyBack CLR L2P InfLoRA AdaptFormer MeLo
Builds on: LoRA PiggyBack CLR DoRA AdaptFormer
Closest papers by meaning
- SD-LoRA: Scalable Decoupled Low-Rank Adaptation for Class Incremental Learning (ICLR 2025 · similarity 0.88)
- CL-LoRA: Continual Low-Rank Adaptation for Rehearsal-Free Class-Incremental Learning (CVPR 2025 · similarity 0.87)
- C-LoRA: Continual Low-Rank Adaptation for Pre-trained Visual Models (NeurIPS 2026 · similarity 0.87)
- MIND: Multi-Task Incremental Network Distillation (AAAI 2024 · similarity 0.86)
- Reinforced Fast Weights via Next-Sequence Prediction (NeurIPS 2026 · similarity 0.86)
- HyperNet Fields: Efficiently Training Hypernetworks without Ground Truth by Learning Weight Trajectories (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.