Research map / Architectures and efficiency / Pruning and efficient tuning

PELA: Learning Parameter-Efficient Models with Low-Rank Approximation

CVPR 2024 · Architectures and efficiency › Pruning and efficient tuning · approach: Low-rank compression and merging · 11 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Compresses pre-trained ViT/ALBEF weights via SVD low-rank factorization, then runs an intermediate pre-training stage with feature distillation and regularized weight perturbation so only the compact low-rank model is needed for downstream fine-tuning.

Problem

Pre-trained large Transformers are expensive to load and fine-tune under limited resources, and existing parameter-efficient methods (adapters, prompt tuning, LoRA) keep the full frozen backbone in memory, so they do not reduce model size.

Approach

Replace each fully connected weight with two SVD-derived low-rank factors, then during an intermediate pre-training stage train only the low-rank model while distilling per-layer token features from the frozen original model and bounding the weight reconstruction error in l-infinity norm; the resulting low-rank model replaces the original for downstream fine-tuning.

Key results

• On ImageNet-1K, PELA on DeiT-Base reaches 81.0% vs 81.8% for the original with about 44M vs 86.6M parameters and 17.0 vs 33.7 GFLOPs.
• On ADE20K semantic segmentation with DeiT-Base, PELA gets 43.2 mIoU vs 45.0 for the original, while LoRA gets 40.6.
• On ALBEF, PELA uses 173M parameters vs 419M for the original and keeps most of the performance (e.g., Flickr30K TR@1 91.6 vs 93.4).

Measured effects reported in the paper

Surprising findings

• Directly applying low-rank approximation to pre-trained weights drops DeiT accuracy from 81% to 61%, showing the decomposition alone is insufficient without intermediate training.
• With a larger batch size enabled by the smaller model, PELA+ on Swin-Base outperforms the original Swin-Base baseline on segmentation (47.99 vs 47.68 mIoU).

Limitations

• Performance drops of roughly 0.6 to 1 point on classification and retrieval, with larger gaps on some tasks such as segmentation for DeiT (45.0 to 43.2 mIoU) and LoRA-level retrieval.
• Uses a single universal compression ratio across all matrix multiplications; dynamic per-layer ratios are left for future work.
• Efficiency gains are computed mainly for matrix multiplications; attention computation is not optimized.

Experimental setup

Datasets: ImageNet-1k ADE20K MSCOCO Flickr30K SNLI-VE VQA RefCOCO+

Benchmarks: ImageNet-1K ADE20K MSCOCO Flickr30K SNLI-VE VQA visual grounding

Metrics: top-1 accuracy mIoU AP box recall@K accuracy GFLOPs parameters GPU memory latency

Baselines: LoRA TinyBERT MaskAlign ToMe Original backbone

Builds on: LoRA TinyBERT MaskAlign ALBEF DeiT

Closest papers by meaning

• RandLoRA: Full rank parameter-efficient fine-tuning of large models (ICLR 2025 · similarity 0.90)
• ExPLoRA: Parameter-Efficient Extended Pre-Training to Adapt Vision Transformers under Domain Shifts (ICML 2025 · similarity 0.89)
• Low-Rank Rescaled Vision Transformer Fine-Tuning: A Residual Design Approach (CVPR 2024 · similarity 0.88)
• SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptation (NeurIPS 2026 · similarity 0.87)
• Efficient Adaptation of Pre-trained Vision Transformer via Householder Transformation (NeurIPS 2024 · similarity 0.87)
• Connecting the Dots: Collaborative Fine-tuning for Black-Box Vision-Language Models (ICML 2024 · similarity 0.87)