Research map / Architectures and efficiency / Pruning and efficient tuning
A geometric framework for momentum-based optimizers for low-rank training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes geometry-aware momentum optimizers (low-rank heavy ball and low-rank Adam) built on dynamical low-rank training that keep weights, gradients and optimizer states jointly low-rank.
Problem
Standard momentum optimizers (heavy ball, Adam) applied to LoRA-style low-rank factors do not follow a gradient flow that converges to a low-rank optimum, and naive DLRT with momentum degrades accuracy.
Approach
Combine dynamical low-rank approximation with momentum: the momentum is kept in the same augmented low-rank basis as the weights, projected onto the new basis after each augmentation-truncation step, and the Adam second moment is projected via the element-wise square root to keep positivity, yielding a geometry-respecting low-rank momentum flow.
Key results
- On GLUE with DeBERTaV3-base, the proposed method reaches a mean score of 88.75 at about 0.89M parameters, the best among compared methods including GeoLoRA (88.19).
- On GPT-2 124M pretraining from scratch (OpenWebText), the method reaches validation loss 3.4642 at 60.61% compression, versus 4.8141 for LoRA pretraining and 3.2313 for the full-rank baseline.
- On VGG16/UCM the method slightly exceeds baseline accuracy (94.61% vs 94.40%) at 95.84% compression, while naive DLRT without momentum projection drops 5 to 13%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the proposed low-rank method instead of LoRA pretraining | GPT-2 pretraining · OpenWebText | validation loss: 7.0242 -> 3.4642 | Helped |
| Use the proposed low-rank method instead of full-rank training | GPT-2 pretraining · OpenWebText | validation loss: 3.2313 -> 3.4642 | Hurt |
| Use proposed method instead of LoRA (r=6) | Llama-2-7b-chat-hf · BoolQ | accuracy: 62.17 % -> 84.09 % | Helped |
| Use proposed method instead of LoRA (r=6) | Llama-2-7b-chat-hf · PIQA | accuracy: 52.18% -> 76.77% | Helped |
| Use proposed method instead of full-rank training | ViT-L.32 · ImageNet-1k | Top-1 accuracy: 74.37 -> 72.27 | Hurt |
| Use proposed method instead of LoRA pretraining | ViT-L.32 · ImageNet-1k | Top-1 accuracy: 63.20 -> 72.27 | Helped |
| Remove momentum-term projection from DLRT (ablation) | DLRT · UCM/CIFAR transfer learning | accuracy: null | Hurt |
| Apply Adam directly to DLRT without the proposed optimizer adaptation (ablation) | ViT-small pretraining · CIFAR-10 | validation accuracy: null | Hurt |
| Use the proposed low-rank method instead of LoRA pretraining | ViT-small pretraining · CIFAR-10 | validation accuracy: null | Helped |
| Use the proposed low-rank method instead of LoRA transfer learning | VGG16 · CIFAR-10 | accuracy: 75.12 -> 79.01 | Helped |
| Use proposed method instead of GeoLoRA | DeBERTaV3-base finetuning · GLUE | mean score: 88.19 -> 88.75 | Helped |
Surprising findings
- Naive DLRT with Adam without projecting optimizer states causes accuracy drops of 5 to 13 percent.
- Standard low-rank gradient flow with momentum can have stationary points that are not low-rank optima, as shown by a counterexample.
Limitations
- Pretraining from scratch shows a moderate accuracy/loss gap to the full-rank baseline (3.4642 vs 3.2313 validation loss).
- The rank-adaptive truncation tolerance must be tuned per experiment via a parameter sweep.
- The theoretical error bound relies on assumptions (bounded, Lipschitz gradients) and an unverified condition on the momentum projection.
Experimental setup
Datasets: UCM CIFAR-10 CIFAR-100 ImageNet-1k GLUE BoolQ PIQA OpenWebText
Benchmarks: UCM CIFAR-10 CIFAR-100 ImageNet-1k GLUE BoolQ PIQA OpenWebText
Metrics: accuracy compression rate validation loss Matthews correlation F1 wall-clock time
Baselines: full-rank training LoRA LoRA pretrain DLRT w/o projection GeoLoRA AdaLoRA DoRA LoRA+ BitFit Houlsby adapter Pfeiffer adapter
Builds on: dynamical low-rank training LoRA GaLore Adam
Closest papers by meaning
- SMMF: Square-Matricized Momentum Factorization for Memory-Efficient Optimization (AAAI 2025 · similarity 0.88)
- Perfect Parallelization in Mini-Batch SGD with Classical Momentum Acceleration (NeurIPS 2026 · similarity 0.88)
- The Marginal Value of Momentum for Small Learning Rate SGD (ICLR 2024 · similarity 0.88)
- Enhancing Optimizer Stability: Momentum Adaptation of The NGN Step-size (NeurIPS 2025 · similarity 0.88)
- On the Performance Analysis of Momentum Method: A Frequency Domain Perspective (ICLR 2025 · similarity 0.87)
- Random Scaling and Momentum for Non-smooth Non-convex Optimization (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.