Research map / Architectures and efficiency / Pruning and efficient tuning

RandLoRA: Full rank parameter-efficient fine-tuning of large models

ICLR 2025 · Architectures and efficiency › Pruning and efficient tuning · approach: LoRA variants and theory · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes RandLoRA, a PEFT method that approximates full-rank weight updates as a sum of fixed random low-rank bases with learned diagonal scalings, and compares it to LoRA, NoLA, and VeRA across vision, vision-language, and commonsense reasoning.

Problem

LoRA's low-rank weight updates limit representation power; it is unclear whether the gap to full fine-tuning comes from too few trainable parameters or from rank deficiency, especially for vision-language models.

Approach

Write the weight update as a sum of n products of non-trainable random low-rank matrices B_j, A_j with learnable diagonal scaling matrices Lambda_j and Gamma_j (sharing a single A), so that the update reaches full rank while training only about d^2/r + d parameters.

Key results

• On CLIP vision-language classification over 22 datasets, RandLoRA-6 (ViT-B/32) reaches 85.98% accuracy versus 83.74% for LoRA rank 32 at equal parameter count.
• On commonsense reasoning with LLaMA3-8B trained on 170k samples, RandLoRA Performant reaches 85.6 average accuracy versus 85.2 for LoRA Performant.
• On DINOv2/CLIP vision backbones, RandLoRA with equal parameters improves over LoRA and bridges the gap to full fine-tuning on CLIP, while LoRA is already close to fine-tuning on DINOv2.

Measured effects reported in the paper

Surprising findings

• On the 15k-sample commonsense setting, RandLoRA and LoRA can overfit when parameters are scaled up, reducing performance even with dropout.
• Sparse ternary random bases at sqrt(D) sparsity perform comparably to dense bases, but accuracy drops at 98-99% sparsity.

Limitations

• Computational overhead from random basis multiplications increases training time, up to about 100% or more for large models.
• Random bases are not optimal; learned or non-random bases could further reduce approximation error.
• Only a small set of configurations and datasets is explored; hybrid LoRA/RandLoRA is left to future work.

Experimental setup

Datasets: Cars DTD EuroSAT GTSRB MNIST RESISC45 SUN397 SVHN CIFAR10 CIFAR100 ImageNet STL10

Benchmarks: 22 image classification datasets (few-shot, 50%, 100%) 8 commonsense reasoning tasks (BoolQ, PIQA, SIQA, HellaSwag, WinoGrande, ARC-c, ARC-e, OBQA) GLUE (appendix) E2E (appendix)

Metrics: accuracy Matthews correlation Pearson correlation VRAM usage training time CKA similarity loss landscape

Baselines: LoRA NoLA VeRA DoRA full fine-tuning Maple + DePT

Builds on: LoRA VeRA NOLA PRANC

Closest papers by meaning

• Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape (ICML 2025 · similarity 0.92)
• The Expressive Power of Low-Rank Adaptation (ICLR 2024 · similarity 0.92)
• SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptation (NeurIPS 2026 · similarity 0.91)
• Computational Limits of Low-Rank Adaptation (LoRA) Fine-Tuning for Transformer Models (ICLR 2025 · similarity 0.91)
• GraLoRA: Granular Low-Rank Adaptation for Parameter-Efficient Fine-Tuning (NeurIPS 2025 · similarity 0.91)
• LoRA-GA: Low-Rank Adaptation with Gradient Approximation (NeurIPS 2024 · similarity 0.90)