Research map / Architectures and efficiency / Pruning and efficient tuning
WST: Wavelet-Based Multi-scale Tuning for Visual Transfer Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes WST, a parameter-efficient tuning method for ViTs that adds a parallel small-patch embedding branch and wavelet fine-tuning blocks that losslessly down-sample its longer token sequence to fuse multi-scale features.
Problem
Transferring large pre-trained ViTs to downstream tasks needing finer-grained, specialized features is hard for PEFT methods, which operate at a single feature scale; smaller patches improve detail but lengthen token sequences and raise computational cost.
Approach
Add a parallel patch embedding branch with half the patch size and quarter the dimension, then use a Haar discrete wavelet transform to losslessly down-sample its 4x longer token sequence to match the backbone length, fuse via addition and two low-rank linear layers, and reconstruct with inverse wavelet transform.
Key results
- On VTAB-1K with ViT-B/16, WST reaches 78.0% average accuracy with 0.08M trainable parameters, about 1.3% above the prior best (DTL, 76.7%).
- On VTAB-1K with Swin-B, WST achieves 78.2% average with 0.11M parameters, above DTL (77.9%) and FacT (77.4%).
- Ablation: wavelet down-sampling (78.0%) beats interpolation down-sampling (77.2%, 0.19M params) and direct reshaping (76.6%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add WST multi-scale tuning | ViT-B/16 · VTAB-1K | average top-1 accuracy: 76.7 -> 78.0 | Helped |
| add WST multi-scale tuning | Swin-B · VTAB-1K | average top-1 accuracy: 77.9 -> 78.2 | Helped |
| add WST multi-scale tuning | ViT-B/16 · domain generalization | top-1 accuracy: 78.3 -> 78.8; 35.4 -> 37.7; 67.8 -> 68.4; 14.0 -> 15.6; 34.4 -> 36.8 | Helped |
| add WST multi-scale tuning | ViT-B/16 · fine-grained visual classification | average top-1 accuracy: 90.7 vs 90.7 | No clear effect |
| set middle dimension r to 1 instead of 2 (ablation) | WST · VTAB-1K (ViT-B/16) | average top-1 accuracy: 78.0 -> 76.4 | Hurt |
| set middle dimension r to 4 instead of 2 (ablation) | WST · VTAB-1K (ViT-B/16) | average top-1 accuracy: 78.0 -> 78.1 | Helped |
| replace multi-scale fusion with a single ViT-B/8 branch (ablation) | WST · VTAB-1K (ViT-B/16) | average top-1 accuracy: 78.0 -> 76.3 | Hurt |
| replace multi-scale fusion with a single ViT-B/16 branch (ablation) | WST · VTAB-1K (ViT-B/16) | average top-1 accuracy: 78.0 -> 75.5 | Hurt |
| replace wavelet down-sampling with direct reshaping (ablation) | WST · VTAB-1K (ViT-B/16) | average top-1 accuracy: 78.0 -> 76.6 | Hurt |
| replace wavelet down-sampling with interpolation (ablation) | WST · VTAB-1K (ViT-B/16) | average top-1 accuracy: 78.0 -> 77.2 | Hurt |
Limitations
- Feature size of the small-scale branch is fixed; no adaptive variable-scale incorporation.
- Only single-level Haar wavelet used; multi-level wavelet and architecture search are left to future work.
Experimental setup
Datasets: VTAB-1K ImageNet-21K ImageNet-1K Aircraft Pets Food-101 Cars Flowers102 CUB-200-2011 NABirds Stanford Dogs
Benchmarks: VTAB-1K Five fine-grained few-shot benchmarks ImageNet-Sketch ImageNet-V2 ImageNet-A ImageNet-R CUB-200-2011 NABirds Oxford Flowers Stanford Dogs Stanford Cars
Metrics: top-1 accuracy trainable parameters (M)
Baselines: Full fine-tuning Linear probing BitFit Adapter LoRA VPT AdaptFormer NOAH FacT E2VPT SA2VP Hydra
Builds on: Adapter LoRA VPT DTL CrossViT
Closest papers by meaning
- Lessons and Insights from a Unifying Study of Parameter-Efficient Fine-Tuning (PEFT) in Visual Recognition (CVPR 2025 · similarity 0.89)
- Visual Fourier Prompt Tuning (NeurIPS 2024 · similarity 0.88)
- ExPLoRA: Parameter-Efficient Extended Pre-Training to Adapt Vision Transformers under Domain Shifts (ICML 2025 · similarity 0.87)
- On the Surprising Effectiveness of Attention Transfer for Vision Transformers (NeurIPS 2024 · similarity 0.87)
- Strong Baselines for Parameter-Efficient Few-Shot Fine-Tuning (AAAI 2024 · similarity 0.87)
- Efficient Adaptation of Pre-trained Vision Transformer via Householder Transformation (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.