Research map / Architectures and efficiency / Pruning and efficient tuning
Time- Memory- and Parameter-Efficient Visual Adaptation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Low-rank Side Adaptation (LoSA), a parallel low-rank mixer network trained on frozen ViT backbone features so no gradients flow through the backbone, improving parameter, training-time and memory efficiency for visual adaptation.
Problem
Parameter-efficient finetuning methods still backpropagate through the whole pretrained backbone, so their training time and memory costs remain high, especially for large vision foundation models and video.
Approach
Keep the backbone fully frozen and train a lightweight parallel side network of low-rank mixer blocks (alternating channel, spatial and temporal token mixing) over the frozen backbone's intermediate activations, with residual connections so it starts as identity over the backbone output.
Key results
- On VTAB-1K with ViT-B, LoSA (r=16) reaches 78.4% average accuracy with 0.19M parameters, above HST (78.1%) with 0.78M parameters.
- On iNaturalist 2018 with ViT-g, LoSA trains 2.35x faster than LoRA while achieving 0.4 points higher accuracy.
- On Kinetics 400, LoSA with a 4B-parameter ViViT-e backbone reaches 86.2% accuracy, above full finetuning of ViViT-H (84.9%), while ST-Adapter and full finetuning cannot scale to ViViT-e.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LoSA with rank 16 instead of HST | ViT-Base adaptation · VTAB-1K | Average accuracy: 78.1 vs 78.4 | Helped |
| Use LoSA with rank 8 instead of rank 16 (ablation) | LoSA on ViT-Base · VTAB-1K | Average accuracy: 78.4 -> 78.3 | Hurt |
| Use LoSA with rank 4 instead of rank 16 (ablation) | LoSA on ViT-Base · VTAB-1K | Average accuracy: 78.4 -> 77.8 | Hurt |
| Scale the frozen backbone from ViViT-H to ViViT-e | LoSA · Kinetics 400 | Accuracy: 80.0 -> 86.2 | Helped |
| Replace a regular transformer adaptor with a low-rank MLP (ablation) | LoSA adaptor on ViT-g · iNaturalist 2018 | Accuracy: 75.0 -> 80.7 | Helped |
| Alternate channel and token mixing with a low-rank mixer instead of a low-rank MLP (ablation) | LoSA adaptor on ViT-g · iNaturalist 2018 | Accuracy: 80.7 -> 81.3 | Helped |
| Use the encoder-block output rather than the MLP output as the backbone activation (ablation) | LoSA on ViT-g · iNaturalist 2018 | Accuracy: 77.7 -> 81.3 | Helped |
| Use the encoder-block output rather than the multihead-attention output as the backbone activation (ablation) | LoSA on ViT-g · iNaturalist 2018 | Accuracy: 79.1 -> 81.3 | Helped |
| Use both MLP and multihead-attention outputs rather than the encoder-block output (ablation) | LoSA on ViT-g · iNaturalist 2018 | Accuracy: 81.0 vs 81.3 | Hurt |
| Add a side-network layer instead of using linear probing (ablation) | LoSA on ViT-g · iNaturalist 2018 | Accuracy: 73.3 -> 77.8 | Helped |
| Increase the side network from 1 to 40 layers (ablation) | LoSA on ViT-g · iNaturalist 2018 | Accuracy: 77.8 -> 80.6 | Helped |
| Use the backbone output rather than the backbone input as the side-network input (ablation) | LoSA on ViT-g · Places365 | Accuracy: 60.9 -> 61.3 | Helped |
Surprising findings
- Prompt tuning can be slower to train than full finetuning because the added prompt tokens lengthen the sequence.
- Adapting large backbones with efficient methods can outperform full finetuning, suggesting large backbones overfit when fully finetuned.
Limitations
- Inference cost is not reduced; the adapted ViViT-e model is slower at inference than baselines due to the large backbone.
- Evaluated mainly on classification tasks; extension to more complex vision tasks is left to future work.
- Linear probing is the only baseline that could scale to ViViT-e, so comparisons at that scale are limited.
Experimental setup
Datasets: VTAB-1K iNaturalist 2018 iNaturalist 2021 Places365 Kinetics 400 ImageNet-21K
Benchmarks: VTAB-1K iNaturalist 2018 iNaturalist 2021 Places365 Kinetics 400
Metrics: accuracy learned parameters training memory (GB) training throughput (img/sec) inference GFLOPs
Baselines: Full finetuning Linear probing BitFit VPT LoRA LST AdaptFormer NOAH HST ST-Adapter
Builds on: LoRA Ladder Side Tuning MLP-Mixer AdaptFormer ST-Adapter
Closest papers by meaning
- Dynamic Tuning Towards Parameter and Inference Efficiency for ViT Adaptation (NeurIPS 2024 · similarity 0.88)
- Memory Efficient Transformer Adapter for Dense Predictions (ICLR 2025 · similarity 0.88)
- Dual Memory Networks: A Versatile Adaptation Approach for Vision-Language Models (CVPR 2024 · similarity 0.87)
- VFM-Adapter: Adapting Visual Foundation Models for Dense Prediction with Dynamic Hybrid Operation Mapping (AAAI 2025 · similarity 0.86)
- Frustratingly Easy Test-Time Adaptation of Vision-Language Models (NeurIPS 2024 · similarity 0.86)
- Efficient Adaptation of Pre-trained Vision Transformer via Householder Transformation (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.