Research map / Architectures and efficiency / Pruning and efficient tuning
Efficient Adaptation of Pre-trained Vision Transformer via Householder Transformation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HTA, a PEFT method that replaces SVD unitary matrices with Householder reflections built from single vectors, with learnable diagonal scaling, plus a rank-1 LoRA term, for adapting pre-trained ViTs and Swin Transformers.
Problem
Low-rank PEFT methods (LoRA, Adapter) use a fixed bottleneck dimensionality that cannot adapt to layer-wise variation in the needed rank, limiting flexibility when adapting pre-trained vision transformers.
Approach
View the adaptation matrix through SVD and replace the left and right unitary matrices with Householder matrices (each derived from one learnable vector), combined with a learnable diagonal scaling matrix per layer, so the effective rank can vary across layers; an additional rank-1 low-rank term is added for robustness.
Key results
- On VTAB-1k with ViT-B/16, HTA reaches 74.7 mean total accuracy with 0.22M params versus RLRR 74.5 with 0.33M.
- With AugReg ViT-B/16 on VTAB-1k, HTA* reaches 75.7 mean total accuracy, the best among compared methods.
- On FGVC, HTA achieves 90.6 mean accuracy with 0.36M params, versus RLRR 90.4 with 0.47M.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use HTA instead of RLRR | ViT-B/16 · VTAB-1k | Mean Total accuracy: 74.5 -> 74.7 | Helped |
| Use HTA instead of RLRR* | ViT-B/16 AugReg · VTAB-1k | Mean Total accuracy: 75.1 -> 75.7 | Helped |
| Use HTA instead of SSF | ViT-B/16 · FGVC | Mean Total accuracy: 90.7 -> 90.6 | Hurt |
| Use HTA* instead of RLRR* | ViT-B/16 AugReg · FGVC | Mean Total accuracy: 91.0 -> 91.1 | Helped |
| Use HTA instead of RLRR | ViT-Large · VTAB-1k | Mean Total accuracy: 75.2 -> 75.4 | Helped |
| Use HTA instead of RLRR | Swin-Base · VTAB-1k | Mean Total accuracy: 73.0 -> 74.2 | Helped |
| Replace LoRA with HTA on Wq and Wv (ablation) | ViT-B/16 · VTAB-1k | Mean Total accuracy: 72.3 -> 72.7 | Helped |
| Add HTA on WFC1 and WFC2 to HTA on Wq and Wv (ablation) | ViT-B/16 · VTAB-1k | Mean Total accuracy: 72.7 -> 73.9 | Helped |
| Replace Adapter on WFC1 and WFC2 with HTA (ablation) | ViT-B/16 · VTAB-1k | Mean Total accuracy: 71.4 -> 73.0 | Helped |
| Add rank-1 low-rank adaptation to HTA (ablation) | ViT-B/16 · VTAB-1k | Mean Total accuracy: 72.0 -> 74.7 | Helped |
| Increase additive adaptation dimension from 1 to 2 (ablation) | HTA · VTAB-1k | Mean Total accuracy: 74.7 -> 74.5 | Hurt |
| Increase additive adaptation dimension from 1 to 4 (ablation) | HTA · VTAB-1k | Mean Total accuracy: 74.7 -> 75.0 | Helped |
Surprising findings
- Householder-only adaptation without the additive low-rank term shows a clear performance drop, indicating a single Householder vector pair is insufficient alone.
Limitations
- Householder matrices derived from a single vector have limited capacity, so an additional rank-1 low-rank matrix is required.
- Gains on FGVC are small because baseline performance is already near saturation.
- Experiments are limited to classification tasks on ViT and Swin backbones.
Experimental setup
Datasets: VTAB-1k CUB-200-2011 NABirds Oxford Flowers Stanford Dogs Stanford Cars ImageNet-21K
Benchmarks: VTAB-1k FGVC
Metrics: top-1 accuracy mean accuracy trainable parameters (M)
Baselines: Full fine-tuning Linear probing Bias VPT LoRA Adapter AdaptFormer FacT ARC RLRR SSF
Builds on: LoRA Adapter AdaptFormer FacT ARC
Closest papers by meaning
- Low-Rank Rescaled Vision Transformer Fine-Tuning: A Residual Design Approach (CVPR 2024 · similarity 0.90)
- ExPLoRA: Parameter-Efficient Extended Pre-Training to Adapt Vision Transformers under Domain Shifts (ICML 2025 · similarity 0.89)
- Canonical Rank Adaptation: An Efficient Fine-Tuning Strategy for Vision Transformers (ICML 2025 · similarity 0.89)
- Bridging The Gap between Low-rank and Orthogonal Adaptation via Householder Reflection Adaptation (NeurIPS 2024 · similarity 0.89)
- Dynamic Tuning Towards Parameter and Inference Efficiency for ViT Adaptation (NeurIPS 2024 · similarity 0.88)
- Sensitivity-Aware Efficient Fine-Tuning via Compact Dynamic-Rank Adaptation (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.