Research map / Architectures and efficiency / Pruning and efficient tuning
Canonical Rank Adaptation: An Efficient Fine-Tuning Strategy for Vision Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CaRA, a PEFT method that tensorises ViT projection weights into an MHA tensor and an FFN tensor and applies a CPD-form low-rank update, outperforming LoRA and tensor-based PEFT baselines on VTAB-1k and FGVC with far fewer parameters.
Problem
LoRA-style matrix low-rank updates ignore the high-dimensional structure of multi-head attention weights in ViTs, limiting expressiveness of parameter-efficient fine-tuning.
Approach
Split the ViT's projection layers into two tensors (MHA Q/K/V stacked across blocks as a 4D tensor; O/up/down FFN layers stacked as a 3D tensor) and fine-tune each with a rank-R Canonical-Polyadic (CP) low-rank update, so the update is a sum of rank-one tensors with only matrix factors.
Key results
- On VTAB-1k with ViT-B/16, CaRA reaches 74.14% overall mean accuracy with 0.06M parameters, versus LoRA at 72.25% (0.29M) and SPT-LoRA at 74.07% (0.54M).
- On FGVC, CaRA averages 90.46% with 0.08M parameters, above SPT-LoRA (90.04%) and VPT-Deep (89.10%).
- On ViT-L few-shot (four datasets), CaRA averages 88.77% versus DoRA at 87.60% and LoRA at 85.98%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace LoRA with CaRA | ViT-B/16 fine-tuning · VTAB-1k | overall mean accuracy: 72.25 -> 74.14 | Helped |
| replace SPT-LoRA with CaRA | ViT-B/16 fine-tuning · VTAB-1k | overall mean accuracy: 74.07 -> 74.14 | Helped |
| replace SPT-LoRA with CaRA | ViT-B/16 fine-tuning · FGVC mean | mean accuracy: 90.04 -> 90.46 | Helped |
| replace DoRA with CaRA | ViT-Large fine-tuning · CIFAR100, Food101, Flowers102, Resisc45 mean | mean accuracy: 87.60 -> 88.77 | Helped |
| use a 5D MHA tensor instead of the 4D MHA tensor (ablation) | CaRA · VTAB-1k Special group | accuracy: 85.134 -> 84.682 | Hurt |
| use a 3D MHA tensor instead of the 4D MHA tensor (ablation) | CaRA · VTAB-1k Special group | accuracy: 85.134 -> 85.061 | Hurt |
| merge MHA and FFN into one 3D tensor (ablation) | CaRA · VTAB-1k Special group | accuracy: 85.134 -> 83.914 | Hurt |
| increase CaRA rank (ablation) | CaRA · VTAB-1k Special group | accuracy | Helped |
| use CaRA instead of LoRA | ViT-Large fine-tuning · CIFAR100, Food101, Flowers102, Resisc45 | training walltime (seconds): 165.756 -> 206.554 | Hurt |
| use CaRA instead of LoRA | ViT-Large fine-tuning · CIFAR100, Food101, Flowers102, Resisc45 | VRAM (GB): 20.108 -> 21.374 | Hurt |
Surprising findings
- CaRA trains slower and uses more VRAM than LoRA (walltime 206.6s vs 165.8s on ViT-L CIFAR100), since the tensor implementation is not optimised.
- Splitting MHA and FFN into separate tensors outperforms a single combined tensor (85.13 vs 83.91 on the Special group).
Limitations
- Training time and memory are higher than LoRA because the tensor-based implementation is not hardware-optimised.
- Gains over SPT-LoRA on the Natural group are not clear; CaRA only matches it while using far fewer parameters.
- Evaluated only on image classification with ViT backbones.
Experimental setup
Datasets: VTAB-1k FGVC (CUB-200-2011, NABirds, Flowers, Stanford Dogs, Stanford Cars) CIFAR100 Food101 Flowers102 Resisc45 FGVC-Aircraft
Benchmarks: VTAB-1k FGVC CIFAR100 Food101 Flowers102 Resisc45
Metrics: top-1 accuracy trainable parameters (M) walltime VRAM
Baselines: LoRA FacT-TT FacT-TK SPT-LoRA NOAH RepAdapter SSF AdaptFormer VPT-Deep DoRA VeRA PiSSA
Builds on: LoRA FacT Canonical-Polyadic Decomposition
Closest papers by meaning
- Efficient Adaptation of Pre-trained Vision Transformer via Householder Transformation (NeurIPS 2024 · similarity 0.89)
- Low-Rank Rescaled Vision Transformer Fine-Tuning: A Residual Design Approach (CVPR 2024 · similarity 0.89)
- RandLoRA: Full rank parameter-efficient fine-tuning of large models (ICLR 2025 · similarity 0.87)
- ExPLoRA: Parameter-Efficient Extended Pre-Training to Adapt Vision Transformers under Domain Shifts (ICML 2025 · similarity 0.86)
- C-LoRA: Continual Low-Rank Adaptation for Pre-trained Visual Models (NeurIPS 2026 · similarity 0.86)
- Sensitivity-Aware Efficient Fine-Tuning via Compact Dynamic-Rank Adaptation (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.