Research map / Architectures and efficiency / Quantization

LRA-QViT: Integrating Low-Rank Approximation and Quantization for Robust and Efficient Vision Transformers

ICML 2025 · Architectures and efficiency › Quantization · approach: Post-training quantization of ViTs

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes LRA-QViT, combining a reparameterizable-branch low-rank approximation (RB-LRA with weight reconstruction initialization), block-level feature knowledge distillation, and an LRA-aware quantization (WADS with per-token activation quantization) to compress ViTs for edge/mobile deployment.

Problem

Low-rank approximation of ViT weights loses information and hurts accuracy, and existing LRA work ignores the subsequent quantization, which produces large activation outliers that degrade quantized accuracy.

Approach

A low-rank residual branch initialized from the SVD-discarded submatrices (weight reconstruction) recovers lost information and is reparameterized into the LRA layer at inference; a weight-aware distribution scaling method and per-token activation quantization then mitigate LRA-induced outliers.

Key results

• Swin-B with RB-LRA+KD reduced parameters by 31.8% and GFLOPs by 30.4% with only a 0.03% top-1 accuracy drop on ImageNet (83.44% vs 83.47%).
• With WADS and INT8 quantization, Swin-B reached 82.97% top-1 (60.1 MB), outperforming NaivePTQ (82.14%) and SmoothQuant (82.76%).
• On mobile and edge devices, RB-LRA+WADS achieved 1.9x-3.2x speedup on Android and 1.5x-2.5x on Jetson Xavier.

Measured effects reported in the paper

Surprising findings

• Random initialization of the low-rank residual branch is not reported to dominate; WR init improves Swin-B from 82.65% to 83.44%, but the authors note per-token quantization is needed because LRA creates token outliers.

Limitations

• Accuracy drops with RB-LRA alone are larger for some models (e.g., DeiT-B 1.92% without KD), and quantization still loses up to 0.56% on DeiT-B.
• Quantization evaluated with static PTQ and only 32 calibration samples; the KD and fine-tuning are only evaluated on ImageNet and a few tasks.
• Latency measured on one Android phone and one Jetson Xavier; LRA-only speedups for the quantized models are compared against FP32 baselines.

Experimental setup

Datasets: ImageNet MSCOCO MSCOCO Keypoint Wikitext-103 LibriSpeech

Benchmarks: ImageNet MSCOCO MSCOCO Keypoint Wikitext-103 LibriSpeech test-clean

Metrics: Top-1 accuracy parameters GFLOPs model size (MB) latency (ms) AP AR perplexity word error rate

Baselines: LRA (naive SVD) PELA AAFM+GFM SmoothQuant Repq-ViT QADS NaivePTQ PTQ4ViT APQ-ViT QDrop I&S-ViT

Builds on: PELA SmoothQuant QADS RepQ-ViT SVD-based low-rank approximation

Closest papers by meaning

• RandLoRA: Full rank parameter-efficient fine-tuning of large models (ICLR 2025 · similarity 0.87)
• GPLQ: A General, Practical, and Lightning QAT Method for Vision Transformers (NeurIPS 2025 · similarity 0.87)
• Bi-ViT: Pushing the Limit of Vision Transformer Quantization (AAAI 2024 · similarity 0.86)
• APHQ-ViT: Post-Training Quantization with Average Perturbation Hessian Based Reconstruction for Vision Transformers (CVPR 2025 · similarity 0.86)
• FIMA-Q: Post-Training Quantization for Vision Transformers by Fisher Information Matrix Approximation (CVPR 2025 · similarity 0.86)
• AIQViT: Architecture-Informed Post-Training Quantization for Vision Transformers (AAAI 2025 · similarity 0.86)