Research map / Architectures and efficiency / Pruning and efficient tuning
Efficient Low-rank Backpropagation for Vision Transformer Adaptation
Yuedong Yang, Hung-Yueh Chiang, Guihong Li, Diana Marculescu, Radu Marculescu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LBP-WHT LP_L1-2 instead of LoRA | LBP-WHT · EfficientFormer-L1 last stage, CIFAR100 | accuracy: 75.61 vs 65.25 | Improved |
| Use LBP-WHT LP_L1-4 instead of LoRA-all | LBP-WHT · EfficientFormer-L1 last stage, CIFAR100 | accuracy: 78.27 vs 76.92 | Improved |
| Use LP_L1-8 rank-36 projection instead of full-rank BP | LBP-WHT · EfficientFormer-L1 last stage, CIFAR100 | accuracy: 79.28 -> 79.34 | Improved |
| Reduce rank from LP_L1-8 to LP_L1-4 (ablation) | LBP-WHT · EfficientFormer-L1 last stage, CIFAR100 | accuracy: 79.34 -> 78.27 | Mixed |
| Reduce rank from LP_L1-4 to LP_L1-2 (ablation) | LBP-WHT · EfficientFormer-L1 last stage, mean accuracy | mean accuracy: 87.76 -> 85.50 | Mixed |
| Use LBP-WHT LP_L1-2 instead of LoRA-all | LBP-WHT · SwinV2-Small last stage, mean accuracy | mean accuracy: 84.86 -> 90.32 | Improved |
| Use LBP-WHT LP_L1-4 instead of LoRA-all | LBP-WHT · Segformer partial, Cityscapes | mIoU: 58.07 -> 60.97 | Improved |
| Use LBP-WHT LP_L1-4 instead of LoRA-all in full training | LBP-WHT · Segformer full training, Cityscapes | mIoU: 65.74 -> 64.72 | Hurt |
| Use a small rank (LP_L1-2) in full training | LBP-WHT · Segformer full training, Cityscapes | mIoU: 67.37 -> 61.57 | Hurt |
| Use LP_L-inf base selection instead of LP_L1 (ablation) | LBP-WHT · EfficientFormer-L1 CIFAR100 | accuracy: 78.27 -> 77.64 | No clear effect |
| Use LHE profiling instead of LP_L1 base selection (ablation) | LBP-WHT · EfficientFormer-L1 CIFAR100 | accuracy: 78.27 -> 78.06 | No clear effect |
| Use LP_L1 low-frequency WHT bases (ablation) | LBP-WHT · EfficientFormer-L1 last attention block | gradient energy preserved | Improved |
Closest papers by meaning
- Efficient Adaptation of Pre-trained Vision Transformer via Householder Transformation (NeurIPS 2024 · similarity 0.88)
- RandLoRA: Full rank parameter-efficient fine-tuning of large models (ICLR 2025 · similarity 0.88)
- Linearly Decomposing and Recomposing Vision Transformers for Diverse-Scale Models (NeurIPS 2024 · similarity 0.87)
- LRA-QViT: Integrating Low-Rank Approximation and Quantization for Robust and Efficient Vision Transformers (ICML 2025 · similarity 0.87)
- RePaViT: Scalable Vision Transformer Acceleration via Structural Reparameterization on Feedforward Network Layers (ICML 2025 · similarity 0.87)
- Low-Rank Rescaled Vision Transformer Fine-Tuning: A Residual Design Approach (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
