Research map / Architectures and efficiency / Pruning and efficient tuning
Structured Unrestricted-Rank Matrices for Parameter Efficient Finetuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SURM, a family of structured unrestricted-rank matrices (circulant, Toeplitz, Kronecker, low displacement rank) as drop-in replacements for LoRA and adapter low-rank matrices for parameter-efficient fine-tuning.
Problem
Low-rank updates in LoRA and adapters constrain expressiveness, while full fine-tuning is costly; the paper asks whether structured matrices that are not low-rank but have linear parameter counts can give better accuracy per parameter.
Approach
Parameterize PEFT updates with low displacement rank matrices (circulant, Toeplitz, Kronecker products) instead of low-rank factors, giving full-rank-capable updates with O(n) parameters and fast FFT-based multiplication; the paper is the first to apply LDRMs to PEFT.
Key results
- On CLIP-ViT, SURM (Circulant) at 0.055M parameters reaches 97.0 on CIFAR10 and 81.1 on SUN397, with the paper reporting 5-7% accuracy gains over LoRA on some image tasks (ViT-B).
- On GLUE with BERT-base, SURM adapters reduce adapter parameters by up to 12x with virtually no quality loss.
- On Synapse segmentation with SAM, SURM (Circulant) reaches 80.11 DSC versus 78.26 for LoRA rank 1 at the same parameter count.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace LoRA updates with SURM matrices | ViT-B/16 image classification · SUN397 and DTD | accuracy | Helped |
| Replace LoRA updates with SURM matrices | CLIP-ViT-B/16 image classification · CIF-10, CIF-100, SUN397, DTD, and STL10 | accuracy | Mixed |
| Use circulant SURM instead of other SURM variants | SURM in low-data vision training · Five image datasets | accuracy | Helped |
| Replace rank-1 LoRA with circulant SURM at matched parameter count | SAM image segmentation · Synapse multi-organ segmentation | DSC: 78.26 -> 80.11 | Helped |
| Use Toeplitz rather than circulant matrices for approximation | Structured matrix approximation · Low-intrinsic-rank matrices | relative Frobenius norm error | Hurt |
| Use Toeplitz rather than circulant matrices for approximation | Structured matrix approximation · Near-low-rank matrices | relative Frobenius norm error | Helped |
| Replace rank-1 LoRA with LDRM matrices at matched parameter count | Pinwheel toy neural network · Pinwheel dataset | training loss | Helped |
| Use SURM instead of LoRA | BERT-based GLUE fine-tuning · GLUE | average performance | Helped |
| Increase the rank parameter r of W(G,H) (ablation) | Structured matrix approximation · Random matrices | relative Frobenius norm error | Mixed |
Surprising findings
- Low-rank matrices were the worst approximators in the PSD approximation experiment, behind Kronecker, circulant and Toeplitz for every trial.
- LoRA learns weights very similar to the pre-trained weights, whereas SURM explores a larger parameter space.
Limitations
- Toeplitz variant is slower than circulant because it needs two fast matrix-vector operations.
- Results on large-scale LLM experiments are limited to a synthetic UUID task on LLaMA-2-7B.
Experimental setup
Datasets: GLUE CIFAR10 CIFAR100 SUN397 DTD STL10 VTAB-1k ImageNet iNat2021 Synapse multi-organ segmentation random UUID pairs
Benchmarks: GLUE CIFAR10 CIFAR100 SUN397 DTD STL10 VTAB-1k ImageNet iNat2021 Synapse multi-organ segmentation
Metrics: accuracy Matthews correlation Dice similarity coefficient relative Frobenius error number of trainable parameters
Baselines: LoRA LoRA-Fix Adapter AdapterDrop KAdaptation AdaptFormer NOAH SSF RepAdapter MoRA SAMed U-Net
Builds on: LoRA Adapters (Houlsby) Compacter Kronecker product adapters
Closest papers by meaning
- RandLoRA: Full rank parameter-efficient fine-tuning of large models (ICLR 2025 · similarity 0.88)
- Computational Limits of Low-Rank Adaptation (LoRA) Fine-Tuning for Transformer Models (ICLR 2025 · similarity 0.88)
- SVFT: Parameter-Efficient Fine-Tuning with Singular Vectors (NeurIPS 2024 · similarity 0.88)
- Sensitivity-Aware Efficient Fine-Tuning via Compact Dynamic-Rank Adaptation (CVPR 2025 · similarity 0.87)
- Be Confident in What You Know: Bayesian Parameter Efficient Fine-Tuning of Vision Foundation Models (NeurIPS 2024 · similarity 0.86)
- Strong Baselines for Parameter-Efficient Few-Shot Fine-Tuning (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.