Research map / Architectures and efficiency / Pruning and efficient tuning
Low-Rank Rescaled Vision Transformer Fine-Tuning: A Residual Design Approach
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RLRR, a residual low-rank rescaling PEFT method for Vision Transformers, motivated by an SVD analysis of existing PEFT methods, and evaluated on VTAB-1k and FGVC image classification.
Problem
Parameter-efficient fine-tuning of pre-trained Vision Transformers must balance retaining the pre-trained model's generalizable representations against adapting to downstream tasks; existing methods offer little guidance on this trade-off.
Approach
Views PEFT methods through the SVD of pretrained weight matrices, then adds a residual term: the frozen matrix plus a learnable row- and column-wise rescaling (s_left ⊙ W ⊙ s_right) and a shift, so the frozen spectrum is kept as a constant base while the scaling adapts it; the update is merged into W after training.
Key results
- On VTAB-1k with ViT-B/16, RLRR reaches 74.5% mean accuracy versus 73.4% for ARC, using 0.33M parameters.
- With the AugReg backbone, RLRR* reaches 75.1% on VTAB-1k versus 74.3% for ARC* and 91.0% mean on FGVC versus 90.7% for SSF and ARC*.
- On ViT-L/16 and ViT-H/14, RLRR reports gains of 2.7% and 2.6% over the latest state of the art on VTAB-1k (per the paper's claim).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RLRR instead of ARC | ViT-B/16 · VTAB-1k | mean accuracy: 73.4 -> 74.5 | Helped |
| Use RLRR* instead of ARC* | AugReg ViT-B/16 · VTAB-1k | mean accuracy: 74.3 -> 75.1 | Helped |
| Use RLRR* instead of SSF | AugReg ViT-B/16 · FGVC | mean accuracy: 90.7 -> 91.0 | Helped |
| Use RLRR instead of SSF | ViT-L/16 · VTAB-1k | mean accuracy: 73.0 -> 75.2 | Helped |
| Use RLRR instead of SSF | ViT-H/14 · VTAB-1k | mean accuracy: 70.4 -> 72.0 | Helped |
| Use RLRR instead of ARC | Swin Transformer · VTAB-1k | mean accuracy: 72.6 -> 73.0 | Helped |
| Add the residual term to left-and-right scaling (ablation) | RLRR · FGVC | mean accuracy: 88.9 -> 90.4 | Helped |
| Add left scaling to right scaling with residual (ablation) | RLRR · FGVC | mean accuracy: 89.7 -> 90.4 | Helped |
| Add right scaling to left scaling with residual (ablation) | RLRR · FGVC | mean accuracy: 89.5 -> 90.4 | Helped |
| Add right scaling to left scaling without residual (ablation) | RLRR · FGVC | mean accuracy: 89.2 -> 88.9 | Hurt |
| Add left scaling to right scaling without residual (ablation) | RLRR · FGVC | mean accuracy: 89.3 -> 88.9 | Hurt |
Surprising findings
- Full fine-tuning on VTAB-1k and FGVC performs worse than many PEFT methods, which the authors attribute to overfitting.
- In the ablation, one-sided rescaling beats two-sided rescaling without the residual term, but the ordering reverses once the residual term is included.
Limitations
- Evaluation is limited to image classification (VTAB-1k and FGVC); no dense prediction or language tasks.
- Gains over strong baselines such as ARC and SSF are often small (e.g., 0.2 to 1.0 points on FGVC), and several comparisons depend on the AugReg backbone.
Experimental setup
Datasets: ImageNet-21k VTAB-1k CUB-200-2011 NABirds Oxford Flowers Stanford Dogs Stanford Cars
Benchmarks: VTAB-1k FGVC (CUB-200-2011, NABirds, Oxford Flowers, Stanford Dogs, Stanford Cars) CIFAR-100 (ablation)
Metrics: top-1 accuracy mean accuracy number of trainable parameters (M)
Baselines: Full fine-tuning Linear probing Adapter Bias VPT-Shallow VPT-Deep LoRA AdaptFormer FacT ARC SSF
Builds on: LoRA SSF VPT AdaptFormer ARC
Closest papers by meaning
- Efficient Adaptation of Pre-trained Vision Transformer via Householder Transformation (NeurIPS 2024 · similarity 0.90)
- Tuning Stable Rank Shrinkage: Aiming at the Overlooked Structural Risk in Fine-tuning (CVPR 2024 · similarity 0.89)
- RandLoRA: Full rank parameter-efficient fine-tuning of large models (ICLR 2025 · similarity 0.89)
- Canonical Rank Adaptation: An Efficient Fine-Tuning Strategy for Vision Transformers (ICML 2025 · similarity 0.89)
- Strong Baselines for Parameter-Efficient Few-Shot Fine-Tuning (AAAI 2024 · similarity 0.88)
- PELA: Learning Parameter-Efficient Models with Low-Rank Approximation (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.