Research map / Architectures and efficiency / Pruning and efficient tuning
StelLA: Subspace Learning in Low-rank Adaptation using Stiefel Manifold
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes StelLA, a three-factor LoRA (USV^T) whose U and V factors are constrained to the Stiefel manifold and optimized with a modular Riemannian wrapper that turns any Euclidean optimizer into a geometry-aware one.
Problem
LoRA lags full fine-tuning, partly because it does not exploit the geometry of low-rank subspaces; SVD-based initializations only affect the start of training and do not optimize subspaces during training.
Approach
Represent the adapter as U S V^T with U and V on Stiefel manifolds (orthonormal columns) and S unconstrained, optimizing U and V via a plug-in procedure: Euclidean optimizer step on Riemannian gradients, tangent projection, and polar retraction, so any Euclidean optimizer can be used without modifying its internals.
Key results
- On commonsense reasoning, StelLA raises LLaMA3-8B average accuracy from 85.40 (best baseline ScaledAdamW) to 86.72 and LLaMA2-7B from 81.00 to 82.33.
- On math and code generation with LLaMA2-7B, StelLA averages 39.30 versus 37.76 for LoRA and 36.61 for DoRA.
- On text-to-image, StelLA achieves the lowest FID on most of the five CivitAI datasets for SD 1.5 and SD 2.0, e.g., 124.46 vs 127.11 (DoRA) on Cyberpunk for SD 1.5.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace LoRA with StelLA | LLaMA2-7B · Commonsense reasoning | average accuracy: 81.00 -> 82.33 | Helped |
| replace LoRA with StelLA | LLaMA3-8B · Commonsense reasoning | average accuracy: 85.40 -> 86.72 | Helped |
| replace LoRA with StelLA | LLaMA2-7B · Math and code generation average | average score: 37.76 -> 39.30 | Helped |
| replace DoRA with StelLA | ViT-Base · Image classification average | accuracy: 90.95 -> 91.20 | Helped |
| replace the product Stiefel geometry with Euclidean geometry (ablation) | StelLA on LLaMA3-8B · Commonsense reasoning | average accuracy: 86.7 -> 84.4 | Hurt |
| replace the product Stiefel geometry with quotient geometry (ablation) | StelLA on LLaMA3-8B · Commonsense reasoning | average accuracy: 86.7 -> 85.7 | Hurt |
| initialize S to zero instead of identity (ablation) | StelLA on LLaMA3-8B · Commonsense reasoning | average accuracy: 86.7 -> 86.5 | Hurt |
| subtract the adapter initialization from the pretrained weight (ablation) | StelLA on LLaMA3-8B · Commonsense reasoning | average accuracy: 86.7 -> 84.2 | Hurt |
| initialize U and V with leading singular vectors (ablation) | StelLA on LLaMA3-8B · Commonsense reasoning | average accuracy: 86.7 -> 86.7 | No clear effect |
| initialize U and V with trailing singular vectors (ablation) | StelLA on LLaMA3-8B · Commonsense reasoning | average accuracy: 86.7 -> 86.6 | No clear effect |
| add gradient scaling for U and V (ablation) | StelLA with AdamW on LLaMA3-8B · Commonsense reasoning | average accuracy: 86.4 -> 86.7 | Helped |
| add parallel transport (ablation) | StelLA on LLaMA3-8B · Commonsense reasoning | average accuracy: 86.7 -> 86.5 | No clear effect |
Surprising findings
- Pseudo-zero initialization (subtracting the adapter init from pretrained weights) performs worst, and the authors find this trick unnecessary for StelLA.
- Parallel transport from Riemannian Adam gives no gain over the proposed Euclidean-optimizer-to-Riemannian conversion.
Limitations
- Small computational overhead versus LoRA from the three-factor formulation, gradient conversion, and retraction.
- Not combined with AdaLoRA-style rank adaptation.
- Not evaluated on other model families (e.g., Mistral, LLaVA) or very large models (LLaMA2-70B).
Experimental setup
Datasets: commonsense reasoning benchmark (8 sub-tasks) MetaMathQA CodeFeedback ImageNet-21K Caltech101 CUB200 Cars196 Flowers102 Food101 OxfordPets Sun397 CIFAR10
Benchmarks: BoolQ PIQA SIQA HellaSwag WinoGrande ARC-e ARC-c OBQA GSM8K MATH HumanEval MBPP
Metrics: accuracy FID CLIP score
Baselines: LoRA DoRA PiSSA OLoRA TriLoRA MoSLoRA ScaledAdamW MiLoRA LoRA-GA
Builds on: LoRA DoRA PiSSA MiLoRA Riemannian Adam rsLoRA
Closest papers by meaning
- SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptation (NeurIPS 2026 · similarity 0.89)
- Riemannian Geometric-based Meta Learning (AAAI 2025 · similarity 0.88)
- Riemannian Preconditioned LoRA for Fine-Tuning Foundation Models (ICML 2024 · similarity 0.88)
- A Stronger Mixture of Low-Rank Experts for Fine-Tuning Foundation Models (ICML 2025 · similarity 0.88)
- PoLAR: Polar-Decomposed Low-Rank Adapter Representation (NeurIPS 2025 · similarity 0.88)
- C-LoRA: Continual Low-Rank Adaptation for Pre-trained Visual Models (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.