Research map / Learning theory and algorithms / Matrix and tensor methods
PoLAR: Polar-Decomposed Low-Rank Adapter Representation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PoLAR, a polar-decomposition-based LoRA reparameterization with two Stiefel-constrained direction matrices and an unconstrained scale matrix, optimized with a landing field instead of retractions, to counter low stable rank of LoRA updates.
Problem
LoRA updates learned by fine-tuning have a low stable rank (near 1 even at rank 32), so the allocated subspace is underutilized and neuron update directions collapse.
Approach
Factor the low-rank update as X Theta Y^T with X and Y on Stiefel manifolds (orthonormal columns) and unconstrained Theta, and fine-tune via a retraction-free landing field so that the update directions remain diverse and the stable rank increases.
Key results
- On GLUE with DeBERTa-v3-base, PoLAR averages 88.94 vs 88.18 for LoRA.
- On commonsense reasoning with Llama-2-7B at rank 16, PoLAR averages 74.00 vs 72.76 for LoRA; at rank 32 it reaches 73.71 vs 72.18.
- On math with Gemma-3-27B-PT, PoLAR reaches 85.67 GSM8K and 42.70 MATH vs LoRA's 85.37 and 41.94; landing is about 3x to 18x faster than retraction in runtime benchmarks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace LoRA with PoLAR at rank 16 | Llama-2-7B · Commonsense reasoning average | accuracy: 72.76 -> 74.00 | Helped |
| replace LoRA with PoLAR at rank 4 | Llama-2-7B · Commonsense reasoning average | accuracy: 72.51 -> 73.26 | Helped |
| replace LoRA with PoLAR at rank 8 | Llama-2-7B · Commonsense reasoning average | accuracy: 72.59 -> 73.51 | Helped |
| replace LoRA with PoLAR at rank 32 | Llama-2-7B · Commonsense reasoning average | accuracy: 72.18 -> 73.71 | Helped |
| replace LoRA with PoLAR | Gemma-3-27B-PT fine-tuned on MetaMathQA · GSM8K | accuracy: 85.37 -> 85.67 | Helped |
| replace LoRA with PoLAR | Gemma-3-27B-PT fine-tuned on MetaMathQA · MATH | accuracy: 41.94 -> 42.70 | Helped |
| replace LoRA with PoLAR | DeBERTa-V3-base · GLUE average | score: 88.18 -> 88.94 | Helped |
| replace LoRA with PoLAR | DeBERTa-V3-base · GLUE MRPC | accuracy: 89.71 -> 89.52 | No clear effect |
| replace LoRA with PoLAR | DeBERTa-V3-base · GLUE QQP | accuracy: 92.47 -> 92.12 | No clear effect |
| use a full unconstrained Θ matrix and Riemannian gradient instead of diagonal Θ and Euclidean gradient (ablation) | PoLAR on Gemma-2-2B · Commonsense reasoning average, rank 4 | accuracy: 71.62 -> 72.45 | Helped |
| use a full unconstrained Θ matrix and Riemannian gradient instead of diagonal Θ and Euclidean gradient (ablation) | PoLAR on Gemma-2-2B · Commonsense reasoning average, rank 8 | accuracy: 72.12 -> 72.69 | Helped |
| use a full unconstrained Θ matrix and Riemannian gradient instead of diagonal Θ and Euclidean gradient (ablation) | PoLAR on Gemma-2-2B · Commonsense reasoning average, rank 16 | accuracy: 71.96 -> 72.75 | Helped |
Surprising findings
- LoRA updates show stable rank as low as 1.06 at nominal rank 32 on Llama-2-7B, and LoRA's stable rank rarely exceeds 2 across nominal ranks.
Limitations
- Evaluated only on NLP tasks with transformer architectures; no vision or other sequence models.
- Model sizes beyond 27B not covered; most model-dataset combinations run with a single seed.
- Extra fine-tuning overhead versus LoRA from the landing field and the r x r matrix.
Experimental setup
Datasets: MetaMathQA GLUE commonsense reasoning suite (BoolQ, PIQA, SIQA, HellaSwag, WinoGrande, ARC-e, ARC-c, OBQA)
Benchmarks: GLUE BoolQ PIQA SIQA HellaSwag WinoGrande ARC-e ARC-c OBQA GSM8K MATH
Metrics: accuracy matthews correlation stable rank runtime
Baselines: LoRA DoRA AdaLoRA GaLore
Builds on: LoRA DoRA AdaLoRA Burer-Monteiro factorization landing algorithm
Closest papers by meaning
- SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptation (NeurIPS 2026 · similarity 0.88)
- StelLA: Subspace Learning in Low-rank Adaptation using Stiefel Manifold (NeurIPS 2025 · similarity 0.88)
- RefLoRA: Refactored Low-Rank Adaptation for Efficient Fine-Tuning of Large Models (NeurIPS 2025 · similarity 0.88)
- Decoupling Angles and Strength in Low-rank Adaptation (ICLR 2025 · similarity 0.87)
- Asymmetry in Low-Rank Adapters of Foundation Models (ICML 2024 · similarity 0.87)
- The Expressive Power of Low-Rank Adaptation (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.