Research map / Architectures and efficiency / Pruning and efficient tuning
Parameter Efficient Fine-tuning via Explained Variance Adaptation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EVA, a LoRA initialization that runs incremental SVD on minibatch activations to set A to top right-singular vectors and redistributes ranks by explained variance, evaluated on NLG, NLU, image classification, and decision making.
Problem
LoRA initialization schemes based on SVD of weights or gradients do not provably maximize the expected gradient signal at the onset of fine-tuning, leading to slow convergence; fixed uniform rank allocation also wastes parameters.
Approach
Initialize LoRA A matrices with the top right-singular vectors of incremental SVD over minibatch activations (variance-optimal, provably maximizing expected squared gradient norm), and redistribute ranks across layers globally by normalized explained variance under a fixed rank budget.
Key results
- EVA applied to Llama-3.1-70B reaches the highest common sense reasoning average (94.5) while reducing trainable parameters by more than 15M relative to LoRA-based methods.
- On GLUE with RoBERTa-Large, EVA achieves the best average of 89.6 versus 89.3 for LoRA and 89.3 for OLoRA; with DeBERTa-v3-Base it reaches 89.9.
- EVA initialization requires only 0.7% of training time (batch size 16) for Llama-2-7B, the fastest among data-driven initializations, reduced to 0.2% with smaller batch size.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Initialize LoRA using activation-variance directions and redistribute ranks | LoRA fine-tuning with RoBERTa Large · GLUE (8 tasks) | Avg: 89.3 -> 89.6 | Helped |
| Initialize LoRA using activation-variance directions and redistribute ranks | LoRA fine-tuning with DeBERTa v3 Base · GLUE (8 tasks) | Avg: 89.6 -> 89.9 | Helped |
| Initialize LoRA using activation-variance directions and redistribute ranks | Llama-2-7B · 8 common-sense reasoning tasks | Avg.: 82.2 -> 83.4 | Helped |
| Combine EVA with DoRA | Llama-2-7B · 8 common-sense reasoning tasks | Avg.: 83.1 -> 84.2 | Helped |
| Initialize LoRA using activation-variance directions and redistribute ranks | Llama-3.1-8B · 8 common-sense reasoning tasks | Avg.: 89.2 -> 89.5 | Helped |
| Combine EVA with DoRA | Llama-3.1-8B · 8 common-sense reasoning tasks | Avg.: 89.3 -> 89.7 | Helped |
| Initialize LoRA using activation-variance directions and redistribute ranks | Gemma-2-9B · 8 common-sense reasoning tasks | Avg.: 92.2 -> 92.5 | Helped |
| Use adaptive rank allocation with EVA initialization (ablation) | EVA fine-tuning · Decision making (Meta-World CW10) | performance | No clear effect |
| Use DoRA with EVA initialization | Decision Transformer fine-tuning · Meta-World CW10 | average performance | Helped |
Surprising findings
- CorDA showed high seed sensitivity and training performance collapsed for certain seeds, despite a hyperparameter search.
- EVA initialization ends up contributing more to the final adapter than LoRA, suggesting it encodes more information at initialization and learns less during training.
Limitations
- EVA assumes access to a fixed static downstream dataset for the SVD initialization.
- Computational overhead of repeated SVD grows at higher ranks (r>=128), and experiments mainly use low rank r=16.
- No single method is best on every task; LoRA outperforms EVA on some specialized image tasks.
Experimental setup
Datasets: BoolQ PIQA SIQA HellaSwag Winogrande ARC-e ARC-c OpenBookQA MetaMathQA GLUE VTAB-1K Meta-World
Benchmarks: BoolQ PIQA SIQA HellaSwag Winogrande ARC-e ARC-c OpenBookQA MATH GSM8K GLUE VTAB-1K
Metrics: accuracy average score Matthews correlation Pearson correlation success rate number of trainable parameters wall-clock overhead
Baselines: LoRA AdaLoRA PiSSA MiLoRA OLoRA LoRA-GA CorDA DoRA FFT
Builds on: LoRA PiSSA LoRA-GA CorDA AdaLoRA DoRA
Closest papers by meaning
- LoRA-GA: Low-Rank Adaptation with Gradient Approximation (NeurIPS 2024 · similarity 0.89)
- Sheared Backpropagation for Fine-tuning Foundation Models (CVPR 2024 · similarity 0.89)
- Parameter-Efficient Fine-Tuning with Discrete Fourier Transform (ICML 2024 · similarity 0.89)
- SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptation (NeurIPS 2026 · similarity 0.89)
- Asymmetry in Low-Rank Adapters of Foundation Models (ICML 2024 · similarity 0.89)
- Beyond Zero Initialization: Investigating the Impact of Non-Zero Initialization on LoRA Fine-Tuning Dynamics (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.