Research map / Architectures and efficiency / Pruning and efficient tuning

AdaRankGrad: Adaptive Gradient Rank and Moments for Memory-Efficient LLMs Training and Fine-Tuning

ICLR 2025 · Architectures and efficiency › Pruning and efficient tuning · approach: LoRA variants and theory · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes AdaRankGrad, which adaptively selects low-rank gradient projections (rank set by an information threshold, found via randomized range finder) for Adam updates, enabling memory-efficient full-parameter LLM pretraining and fine-tuning.

Problem

Memory and compute cost of training and fine-tuning LLMs, where LoRA restricts search to a low-rank subspace and GaLore uses fixed projection rank and fixed update intervals.

Approach

Proves gradient effective rank decays over training toward rank one for reversible networks, and uses this by adaptively choosing per-layer projection rank via an information threshold, updating subspaces when projected gradients converge, and transforming Adam moments across subspace changes.

Key results

• On GLUE with RoBERTa-base at initial rank 4, AdaRankGrad reports 202M memory versus 747M for full fine-tuning, with average accuracy comparable to or above full fine-tuning (e.g., RTE 81.23 vs 79.42).
• Pre-training LLaMA 1B on C4 gives validation perplexity 14.71 for AdaRankGrad versus 15.56 for full-rank and 15.64 for GaLore, with 3.62G optimizer/parameter memory versus 7.80G full-rank.
• Ablations show fixing the subspace update interval or fixing the rank to 4 yields results close to the full AdaRankGrad (e.g., CoLA 61.2 and 61.12 vs 61.4).

Measured effects reported in the paper

Surprising findings

• Reported memory numbers for the GLUE table are inconsistent with the stated fixed-rank GaLore/LoRA comparisons in some entries, and the 7B C4 table shows AdaRankGrad memory of 16.4G versus 18G for GaLore.
• Several GLUE scores for AdaRankGrad are slightly below GaLore or full fine-tuning (e.g., MNLI 86.6 vs 87.18 full fine-tuning).

Limitations

• Theoretical rank-decay result is proven only for reversible layers under vanilla SGD; RoBERTa experiments involve non-reversible layers.
• Convergence analysis relies on smoothness and bounded gradient assumptions and uses SGD rather than Adam.
• Adaptive rank selection may add training time from the rank search.

Experimental setup

Datasets: GLUE C4 Geneformer cardiomyopathy cell classification data

Benchmarks: GLUE C4 validation perplexity

Metrics: accuracy Matthew's correlation Pearson correlation F1 perplexity memory usage Macro-F1

Baselines: Full fine-tuning LoRA GaLore ReLoRA 8-bit Adam Low-Rank

Builds on: GaLore LoRA Adam ReLoRA

Closest papers by meaning

• Fira: Can We Achieve Full-rank Training of LLMs Under Low-rank Constraint? (NeurIPS 2025 · similarity 0.92)
• GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection (ICML 2024 · similarity 0.92)
• Gradient Weight-normalized Low-rank Projection for Efficient LLM Training (AAAI 2025 · similarity 0.91)
• GoRA: Gradient-driven Adaptive Low Rank Adaptation (NeurIPS 2025 · similarity 0.91)
• AltLoRA: Towards Better Gradient Approximation in Low-Rank Adaptation with Alternating Projections (NeurIPS 2025 · similarity 0.91)
• Mini-batch Coresets for Memory-efficient Language Model Training on Data Mixtures (ICLR 2025 · similarity 0.91)