Research map / Architectures and efficiency / Pruning and efficient tuning
VeRA: Vector-based Random Matrix Adaptation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VeRA, which freezes a single shared pair of random low-rank matrices and trains only small scaling vectors, cutting trainable parameters roughly 10x versus LoRA while matching its accuracy on GLUE, E2E, ViT and instruction tuning.
Problem
LoRA still requires substantial trainable parameters and storage per adapted model, which becomes costly when scaling to very large models or serving many per-user or per-task finetuned versions.
Approach
Freeze a single pair of random projection matrices shared across all layers and learn only per-layer scaling vectors (b and d) applied as diagonal matrices, so the update is Λb B Λd A x; the random matrices are regenerated from an RNG seed, so only the vectors need storing.
Key results
- On GLUE with RoBERTa-large, VeRA reaches 87.8 average, matching LoRA's 87.8 with 0.061M vs 0.8M trainable parameters.
- On E2E with GPT-2 Medium, VeRA achieves BLEU 70.1 vs LoRA 68.9 using 0.098M vs 0.35M parameters.
- On MT-Bench with Llama2 13B, VeRA scores 5.93 vs LoRA 5.77 using 2.4M vs 250.3M parameters.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VeRA instead of LoRA | RoBERTa-large · GLUE Avg. | average score: 87.8 -> 87.8 | No clear effect |
| Use VeRA instead of LoRA | GPT-2 Medium · E2E | BLEU: 68.9 -> 70.1 | Helped |
| Use VeRA instead of LoRA | GPT-2 Large · E2E | BLEU: 70.1 -> 70.3 | Helped |
| Use VeRA instead of LoRA | Llama and Llama2 models · MT-Bench | GPT-4 score: 5.03 -> 4.77; 5.31 -> 5.22; 5.19 -> 5.08; 5.77 -> 5.93 | Mixed |
| Use VeRA instead of LoRA | ViT-B and ViT-L · Image classification (CIFAR100, Food101, Flowers102, RESISC45) | accuracy: 85.9 -> 84.8; 98.8 -> 99.0; 87.0 -> 87.5; 79.5 -> 79.2 | Mixed |
| Add the b scaling vector (ablation) | VeRA · MRPC, RoBERTa-large | score: 89.7 -> 90.5 | Helped |
| Add the b scaling vector (ablation) | VeRA · RTE, RoBERTa-large | accuracy: 67.0 -> 85.8 | Helped |
| Add the d scaling vector (ablation) | VeRA · MRPC, RoBERTa-large | score: 81.6 -> 90.5 | Helped |
| Add the d scaling vector (ablation) | VeRA · RTE, RoBERTa-large | accuracy: 64.3 -> 85.8 | Helped |
| Replace Kaiming normal with Kaiming uniform matrix initialization (ablation) | VeRA · MRPC, RoBERTa-large | score: 90.0_{\pm1.1} -> 90.5_{\pm0.7} | No clear effect |
| Replace uniform_[0.0,0.1] with Kaiming uniform matrix initialization (ablation) | VeRA · RTE, RoBERTa-large | accuracy: 53.1_{\pm0.8} -> 85.8_{\pm0.7} | Helped |
| Initialize d to 10^{-1} instead of 1.0 (ablation) | VeRA · MRPC, RoBERTa-large | score: 70.3_{\pm1.2} -> 90.5_{\pm0.7} | Helped |
Surprising findings
- On Llama 7B instruction tuning, VeRA scores lower than LoRA on MT-Bench (4.77 vs 5.03), while on Llama2 13B VeRA scores higher (5.93 vs 5.77).
- Sharing the random matrices across layers gives performance essentially identical to using unique random matrices per layer.
Limitations
- Evaluation limited to language and vision Transformer architectures; applicability to other architectures is untested.
- Hyperparameters were not grid-searched due to compute limits; MNLI and QQP were omitted from GLUE.
- Performance is inconsistent on some settings, e.g. lower MT-Bench score than LoRA for Llama 7B.
Experimental setup
Datasets: GLUE E2E Alpaca cleaned CIFAR100 Food101 Flowers102 RESISC45 ImageNet-21k
Benchmarks: GLUE E2E MT-Bench CIFAR100 Food101 Flowers102 RESISC45 Vicuna Eval
Metrics: accuracy Matthews correlation Pearson correlation BLEU NIST METEOR ROUGE-L CIDEr GPT-4 judge score trainable parameters
Baselines: LoRA LoRA-FA BitFit Adapter (H/L/P/D) AdaLoRA DyLoRA Full finetuning Head-only
Builds on: LoRA AdaLoRA Kaiming initialization intrinsic dimensionality
Closest papers by meaning
- SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptation (NeurIPS 2026 · similarity 0.90)
- Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape (ICML 2025 · similarity 0.90)
- Uni-LoRA: One Vector is All You Need (NeurIPS 2025 · similarity 0.90)
- Dynamic Low-Rank Sparse Adaptation for Large Language Models (ICLR 2025 · similarity 0.89)
- Batched Low-Rank Adaptation of Foundation Models (ICLR 2024 · similarity 0.89)
- AltLoRA: Towards Better Gradient Approximation in Low-Rank Adaptation with Alternating Projections (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.