Research map / Architectures and efficiency / Pruning and efficient tuning
Efficient Stitchable Task Adaptation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ESTA, a parameter-efficient, one-stage framework that adapts SN-Net stitches to downstream tasks using shared LoRA updates, stitch-specific biases, and importance-based stitch sampling.
Problem
Adapting SN-Net (stitchable networks from a pre-trained model family) to downstream tasks is memory-, storage- and time-expensive, requiring a multistage process and sub-optimal Pareto-frontier stitches, when diverse resource budgets must be served.
Approach
Combine stitch-agnostic LoRA updates (on attention and stitching layers) with stitch-specific bias terms for parameter-efficient stitch fine-tuning, train anchors and stitches simultaneously in one stage, and sample stitches with probabilities from SNIP-based moving-average importance scores, reusing those scores to select deployment stitches.
Key results
- On FGVC and VTAB-1k tasks ESTA stitches give smooth FLOPs-accuracy curves that substantially outperform SN-Net adaptation with far fewer trainable parameters (ESTA 4.6M vs SN-Net 124.2M trainable parameters).
- Deployment takes 5.0 GPU hours for ESTA versus 19.3 for SN-Net on a single RTX 3090 when stitching ViT-Ti/S/B.
- Stitched LLaMA (6.9B to 13B) obtains response quality between Alpaca-LoRA 7B (100%) and 13B (123%) on Vicuna Bench, e.g., 115% at 11.5B parameters.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use stitch-agnostic LoRA in self-attention layers instead of full fine-tuning (ablation) | ESTA · Five FGVC tasks | accuracy | Helped |
| Use low-rank updates for stitching layers (ablation) | ESTA with PST · Five FGVC tasks | accuracy | Hurt |
| Add stitch-specific bias terms (ablation) | ESTA with PST · Five FGVC tasks | accuracy | Helped |
| Add stitch-specific bias terms rather than increasing the LoRA rank to the same parameter budget (ablation) | ESTA with PST · Five FGVC tasks | accuracy | Helped |
| Use task-specific importance-based stitch sampling instead of uniform sampling (ablation) | ESTA · Five FGVC tasks | accuracy | Helped |
| Simultaneously adapt and stitch anchors instead of adapting anchors first (ablation) | ESTA · Five FGVC tasks | accuracy | Helped |
| Use Adapter or Adaptformer rather than LoRA (ablation) | ESTA · Five FGVC tasks | accuracy | No clear effect |
| Vary the LoRA rank in self-attention layers (ablation) | ESTA · Five FGVC tasks | accuracy | No clear effect |
| Stitch LLaMA-7B and LLaMA-13B to create models of intermediate sizes | Stitched LLaMA · Vicuna Bench | relative response quality: 100 -> 113; 100 -> 114; 100 -> 115; 100 -> 120 | Helped |
Surprising findings
- Some stitches outperform individually fine-tuned ViT-S/B anchors with comparable FLOPs, which the authors attribute to weight-sharing regularization.
- Direct SN-Net adaptation (adapt anchors individually then stitch) yields poor results, with individually fine-tuned larger anchors likely overfitting.
Limitations
- Experiments limited to visual recognition and instruction tuning; dense prediction and multi-modal tasks not explored.
- Relies on stitching a fixed model family (ViT-Ti/S/B, LLaMA-7B/13B).
Experimental setup
Datasets: ImageNet-21k ImageNet-1k NABirds CUB-200-2011 Stanford Cars Stanford Dogs Oxford Flowers CIFAR-100 VTAB-1k Alpaca 52K instruction data
Benchmarks: NABirds CUB-200-2011 Stanford Cars Stanford Dogs Oxford Flowers CIFAR-100 VTAB-1k Vicuna Bench
Metrics: top-1 accuracy FLOPs-accuracy trade-off relative response quality (%) trainable parameters fine-tuning memory deployment GPU hours
Baselines: SN-Net LoRA fine-tuning of individual anchors Alpaca-LoRA 7B Alpaca-LoRA 13B Adapter AdaptFormer
Builds on: SN-Net LoRA SNIP Stitchable Neural Networks
Closest papers by meaning
- Building Variable-Sized Models via Learngene Pool (AAAI 2024 · similarity 0.88)
- Reconstructing the Image Stitching Pipeline: Integrating Fusion and Rectangling into a Unified Inpainting Model (NeurIPS 2024 · similarity 0.84)
- Implicit Modeling for Transferability Estimation of Vision Foundation Models (NeurIPS 2025 · similarity 0.84)
- SURGEON: Memory-Adaptive Fully Test-Time Adaptation via Dynamic Activation Sparsity (CVPR 2025 · similarity 0.83)
- How Not to Stitch Representations to Measure Similarity: Task Loss Matching Versus Direct Matching (AAAI 2025 · similarity 0.83)
- Time- Memory- and Parameter-Efficient Visual Adaptation (CVPR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.