Research map / Architectures and efficiency / Pruning and efficient tuning

Uni-LoRA: One Vector is All You Need

NeurIPS 2025 · Architectures and efficiency › Pruning and efficient tuning · approach: LoRA variants and theory · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Uni-LoRA, a unified projection view of LoRA variants, and a frozen, isometric random one-hot-column-normalized projection that lets a single trainable vector of length d reconstruct all LoRA parameters.

Problem

Existing LoRA parameter-reduction variants (Tied-LoRA, VeRA, LoRA-XS, VB-LoRA) use layer-local or non-isometric projections, limiting parameter efficiency and sharing across layers; a unified view and a more efficient, geometry-preserving reduction are needed.

Approach

View LoRA parameters as a D-dimensional vector and all LoRA variants as a fixed or learned projection P from a d-dimensional subspace; propose a globally random uniform projection (each row one-hot at a random slot, column-normalized by 1/sqrt(n_j)) that is provably isometric, is O(D) to apply, and needs only one trainable vector.

Key results

• On GLUE with RoBERTa-large, Uni-LoRA reaches 88.3 average with 0.023M trainable parameters, versus 87.8 for LoRA (0.786M) and 88.2 for VB-LoRA (0.162M).
• On GSM8K/MATH with Gemma-7B, Uni-LoRA (0.52M params) scores 75.59 on GSM8K and 28.94 on MATH, above LoRA's 74.90 and 31.28 only on GSM8K; it trails LoRA on MATH.
• On ViT-Base vision tasks, Uni-LoRA (72K params) averages 85.15 versus 77.75 for FourierFT with 72K params.

Measured effects reported in the paper

Surprising findings

• A single random, frozen, isometric projection of the full LoRA parameter space matches learned-projection VB-LoRA on GLUE despite having far fewer trainable parameters.
• Uni-LoRA with uniform global projection outperforms a non-uniform (A/B split) projection and a layer-wise local projection on the controlled ablations.

Limitations

• Only evaluated on small to medium scale NLP and CV benchmarks due to limited compute, as the authors state.
• On MATH with Gemma-7B, Uni-LoRA (28.94) is below LoRA (31.28) and for some comparisons it trails baselines such as VeRA on Mistral MATH.

Experimental setup

Datasets: GLUE SST-2 MRPC CoLA QNLI RTE STS-B MetaMathQA GSM8K MATH Cleaned Alpaca MT-Bench

Benchmarks: GLUE GSM8K MATH MT-Bench OxfordPets StanfordCars CIFAR10 DTD EuroSAT FGVC RESISC45 CIFAR100

Metrics: accuracy Matthews correlation Pearson correlation MT-Bench score training time GPU memory

Baselines: LoRA Tied-LoRA VeRA VB-LoRA LoRA-XS FourierFT Fastfood projection Full-FT

Builds on: LoRA Tied-LoRA VeRA VB-LoRA FourierFT Fastfood

Closest papers by meaning

• Beyond Higher Rank: Token-wise Input-Output Projections for Efficient Low-Rank Adaptation (NeurIPS 2025 · similarity 0.94)
• AuroRA: Breaking Low-Rank Bottleneck of LoRA with Nonlinear Mapping (NeurIPS 2025 · similarity 0.94)
• VB-LoRA: Extreme Parameter Efficient Fine-Tuning with Vector Banks (NeurIPS 2024 · similarity 0.94)
• LoRA-One: One-Step Full Gradient Could Suffice for Fine-Tuning Large Language Models, Provably and Efficiently (ICML 2025 · similarity 0.93)
• Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuning (NeurIPS 2026 · similarity 0.93)
• GoRA: Gradient-driven Adaptive Low Rank Adaptation (NeurIPS 2025 · similarity 0.93)