Research map / Architectures and efficiency / Pruning and efficient tuning

NOLA: Compressing LoRA using Linear Combination of Random Basis

ICLR 2024 · Architectures and efficiency › Pruning and efficient tuning · approach: LoRA variants and theory · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes NOLA, which reparameterizes LoRA's low-rank factors as linear combinations of frozen random basis matrices and trains only the mixture coefficients, breaking LoRA's rank-one parameter floor.

Problem

LoRA's parameter count is lower-bounded by rank one and is tied to layer shape and rank, limiting compression when storing many task-specific fine-tuned LLMs.

Approach

Re-parameterize LoRA's A and B matrices as linear combinations of pseudo-random fixed basis matrices (seeded), optimizing only the mixture coefficients; this decouples trainable parameter count from rank and layer dimensions, and allows coefficients to be quantized to 4 bits.

Key results

• On E2E with GPT-2 L, NOLA with 0.036M trainable parameters reaches BLEU 70.1, versus 70.4 for LoRA rank 4 with 0.77M parameters (about 20x fewer).
• On LLaMA-2 70B (8-bit), NOLA uses 0.57M trainable parameters (95% reduction vs rank-one LoRA at 12.94M) with MMLU 69.4 vs 69.5 for LoRA.
• NOLA with 3-bit quantization-aware training on GPT-2 L loses only ~0.3 BLEU (70.14) while LoRA rank 1 drops ~2.8 BLEU (67.08).

Measured effects reported in the paper

Surprising findings

• NOLA with rank 8 at equal parameter count is reported as matching or beating LoRA, and the paper notes that rank had little effect on NOLA performance when parameter count is held fixed.

Limitations

• Trainable-parameter savings do not reduce inference-time compute, and the paper states it does not claim accuracy gains over baselines.
• Effect of rank is not rigorously studied and is left as future work.
• Non-shared random basis is slightly slower than LoRA in training time.

Experimental setup

Datasets: E2E NLG Challenge DART WebNLG Alpaca MMLU CIFAR-10 CIFAR-100 CUB-200-2011 Caltech-101 Aircraft Food101 Pets

Benchmarks: E2E NLG Challenge DART WebNLG Alpaca MMLU CIFAR-10 CIFAR-100 CUB-200-2011 Caltech-101 Aircraft Food101 Pets

Metrics: BLEU NIST METEOR ROUGE-L CIDEr TER training loss validation loss MMLU accuracy top-1 accuracy

Baselines: LoRA Adapter PreLayer Full fine-tuning QLoRA PRANC HashedNet Linear probe Nearest neighbor

Builds on: LoRA PRANC QLoRA Straight-Through Estimator

Closest papers by meaning

• Uni-LoRA: One Vector is All You Need (NeurIPS 2025 · similarity 0.93)
• Dynamic Low-Rank Sparse Adaptation for Large Language Models (ICLR 2025 · similarity 0.92)
• Train Small, Infer Large: Memory-Efficient LoRA Training for Large Language Models (ICLR 2025 · similarity 0.91)
• BSLoRA: Enhancing the Parameter Efficiency of LoRA with Intra-Layer and Inter-Layer Sharing (ICML 2025 · similarity 0.91)
• Merging LoRAs like Playing LEGO: Pushing the Modularity of LoRA to Extremes Through Rank-Wise Clustering (ICLR 2025 · similarity 0.91)
• Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape (ICML 2025 · similarity 0.91)