Research map / Architectures and efficiency / Pruning and efficient tuning
Dynamic Tuning Towards Parameter and Inference Efficiency for ViT Adaptation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Dynamic Tuning (DyT), a PEFT method for ViTs that uses a learned token dispatcher to skip the MLP block for uninformative tokens plus an MoE-adapter, reducing inference FLOPs while improving VTAB-1K accuracy.
Problem
Existing parameter-efficient fine-tuning methods for vision transformers reduce trainable parameters but do not reduce inference computation, which is costly for large ViT models adapted to downstream tasks.
Approach
A per-block token dispatcher (a single linear projection with Gumbel-sigmoid during training) selects which tokens pass through the frozen block (mainly MLP), while all tokens go through a parallel lightweight adapter; an MoE-adapter with input-dependent expert weights merged into one adapter adds capacity at no extra FLOPs.
Key results
- On VTAB-1K, DyT (r=0.5) reaches 77.14% group-mean accuracy with 12.54 GFLOPs (about 71% of ViT-B's 17.58) and 0.16M extra parameters, versus 74.75% for AdaptFormer at 17.61 GFLOPs.
- DyT with r=0.5 gives 912 img/s on V100 vs 806 for full tuning and LoRA, showing measured speedup.
- On K400 and SSv2, DyT with 4-expert MoE-adapter reaches 75.43% and 46.56%, with 12.29 GFLOPs on CIFAR-100 setting.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DyT with token dispatching before MLP rather than attention dispatch (ablation) | ViT-B/16 · CIFAR-100 | accuracy: 84.58 -> 91.37 | Helped |
| Use DyT with token dispatching before both attention and MLP rather than MLP dispatch (ablation) | ViT-B/16 · CIFAR-100 | accuracy: 91.37 -> 90.03 | Hurt |
| Add a 4-expert MoE adapter (ablation) | DyT · K400 | accuracy: 75.28 -> 75.43 | Helped |
| Add a 4-expert MoE adapter (ablation) | DyT · CIFAR-100 | accuracy: 91.37 -> 91.01 | Hurt |
| Remove the distillation loss (ablation) | DyT · VTAB-1K | accuracy: 77.14 -> 76.70 | Hurt |
| Remove the complete-model loss and distillation loss (ablation) | DyT · VTAB-1K | accuracy: 77.14 -> 75.73 | Hurt |
| Increase the activation rate from r=0.7 to r=0.9 (ablation) | DyT · VTAB-1K | Group Mean accuracy: 77.57 -> 77.30 | Hurt |
| Add ToMe token merging to DyT r=0.5 (ablation) | DyT r=0.5 · VTAB-1K | Throughput (img/s), V100: 912.39 -> 1114.70 | Helped |
| Add ToMe token merging to DyT r=0.5 (ablation) | DyT r=0.5 · VTAB-1K | accuracy: 77.14 -> 76.60 | Hurt |
| Dispatch tokens at token level with the learned token dispatcher instead of random dispatch (ablation) | DyT · K400 | accuracy: 68.27 -> 75.28 | Helped |
| Use adapter bottleneck dimension 4 rather than 64 (ablation) | DyT · CIFAR-100 | accuracy: 91.37 -> 91.57 | Helped |
| Set Gumbel-Sigmoid temperature to 1 instead of 5 (ablation) | DyT · CIFAR-100 | accuracy: 91.37 -> 91.61 | Helped |
Surprising findings
- Dropping the MLP block of the final ViT layer entirely (0% activation) on CIFAR-100, SVHN, and K400 did not hurt accuracy.
- Dynamic-Full (full fine-tuning with a token dispatcher) became unstable and dropped significantly on some datasets such as CIFAR-100 and Food-101.
Limitations
- Designed only for vision tasks; not yet extended to multimodal or large language model settings.
- Training cost is about 1.8x full-model training when using the complete-model teacher distillation.
- Performance on object detection (COCO) lags full tuning substantially.
Experimental setup
Datasets: VTAB-1K CIFAR-100 SVHN Food-101 Air Pet Car Kinetics-400 Something-Something V2 ADE20K COCO-Stuff COCO
Benchmarks: VTAB-1K CIFAR-100 SVHN Food-101 Air Pet Car Kinetics-400 Something-Something V2 ADE20K COCO-Stuff COCO
Metrics: accuracy mIoU box mAP mask mAP FLOPs (G) parameters (M) throughput (img/s)
Baselines: Full tuning Linear probe AdaptFormer LoRA VPT NOAH ConvPass Res-Tuning SSF DynamicViT EViT ToMe
Builds on: AdaptFormer DynamicViT Gumbel-Softmax mixture of experts
Closest papers by meaning
- Efficient Adaptation of Pre-trained Vision Transformer via Householder Transformation (NeurIPS 2024 · similarity 0.88)
- Time- Memory- and Parameter-Efficient Visual Adaptation (CVPR 2024 · similarity 0.88)
- DA-VPT: Semantic-Guided Visual Prompt Tuning for Vision Transformers (CVPR 2025 · similarity 0.87)
- Rethinking Token Reduction with Parameter-Efficient Fine-Tuning in ViT for Pixel-Level Tasks (CVPR 2025 · similarity 0.87)
- Memory Efficient Transformer Adapter for Dense Predictions (ICLR 2025 · similarity 0.87)
- Strong Baselines for Parameter-Efficient Few-Shot Fine-Tuning (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.