Research map / Architectures and efficiency / Pruning and efficient tuning

VB-LoRA: Extreme Parameter Efficient Fine-Tuning with Vector Banks

NeurIPS 2024 · Architectures and efficiency › Pruning and efficient tuning · approach: LoRA variants and theory · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes VB-LoRA, a PEFT method that reparameterizes LoRA's rank-one components into sub-vectors composed from a single globally shared vector bank via a differentiable top-k admixture, achieving extreme parameter efficiency.

Problem

LoRA and its variants still incur substantial storage and transmission costs when customizing LLMs per user or task, and existing sharing schemes (LoRA local, tied/VeRA across layers) do not share parameters across matrix dimensions, modules and layers at fine granularity.

Approach

Divide-and-share: LoRA's rank-one vectors are split into equal-size sub-vectors, and each sub-vector is a sparse top-k softmax combination of vectors from one global vector bank shared across modules and layers; the logits are learned end-to-end, and only the bank, top-k indices and weights are stored.

Key results

• On RoBERTa-large (query and value), VB-LoRA reaches 88.2 GLUE average with 0.024M parameters versus 87.8 for LoRA with 0.786M.
• On Llama2 13B instruction tuning, VB-LoRA scores 6.31 on MT-Bench with 1.1M stored parameters versus 6.13 for the authors' LoRA run with 250.3M.
• VB-LoRA uses 0.4% of LoRA's stored parameters on Llama2 13B while achieving superior results, per the abstract.

Measured effects reported in the paper

Surprising findings

• Noisy Top-k, Gumbel-Softmax and Straight-Through Gumbel-Softmax selection underperform Top-k and Select All substantially on CoLA, which the authors attribute to noise disrupting the vector bank.
• With k=1 the selected vectors remain largely unchanged during training, and k=1 performs significantly worse than k=2 on CoLA.

Limitations

• Experiments limited to text-only, English, and LoRA-only settings.
• Exploration of vector bank size and vector length is limited to a small range of configurations.
• Training incurs computational overhead from sub-vector decomposition (about 15-20% more time than LoRA).

Experimental setup

Datasets: GLUE E2E Cleaned Alpaca MetaMathQA

Benchmarks: GLUE E2E MT-Bench GSM8K MATH

Metrics: accuracy Matthews correlation Pearson correlation BLEU NIST METEOR ROUGE-L CIDEr MT-Bench score stored parameter count

Baselines: LoRA VeRA Tied-LoRA LoRA-XS full fine-tuning Noisy Top-k Gumbel-Softmax Straight-Through Gumbel-Softmax

Builds on: LoRA VeRA Tied-LoRA Noisy Top-k gating QLoRA

Closest papers by meaning

• Uni-LoRA: One Vector is All You Need (NeurIPS 2025 · similarity 0.94)
• HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuning (NeurIPS 2024 · similarity 0.92)
• Beyond Higher Rank: Token-wise Input-Output Projections for Efficient Low-Rank Adaptation (NeurIPS 2025 · similarity 0.91)
• GoRA: Gradient-driven Adaptive Low Rank Adaptation (NeurIPS 2025 · similarity 0.91)
• Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuning (NeurIPS 2026 · similarity 0.91)
• LoRA-Pro: Are Low-Rank Adapters Properly Optimized? (ICLR 2025 · similarity 0.91)