Research map / Architectures and efficiency / Pruning and efficient tuning
LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statistics
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LDAdam, a memory-efficient Adam variant that runs adaptive updates in a low-rank subspace of gradients and optimizer states, using projection-aware state updates and generalized error feedback, with convergence proofs.
Problem
Adam's optimizer states double memory relative to model size; low-rank projection-based optimizers (e.g., GaLore) lose information when the subspace changes and lack convergence guarantees under standard assumptions.
Approach
Projection-aware update rule transports first and second moment estimates between successive low-rank subspaces (via U_t^T U_{t-1}), computed from a block power iteration on an interpolation of the gradient and momentum, plus a generalized error feedback buffer that compensates projection error in both gradients and optimizer states, stored in the gradient buffer to add no extra memory.
Key results
- On RoBERTa-base GLUE fine-tuning, LDAdam (rank 8) averages 86.48 vs Adam 86.28 and GaLore 84.73.
- On C4 pre-training, Llama 130M validation perplexity is 22.65 with LDAdam at rank 256 vs 24.64 for Adam and 26.04 for GaLore.
- On Llama-2 7B GSM8K fine-tuning, LDAdam at rank 512 reaches 35.86% accuracy vs Adam 34.72% with peak memory 35.58 GB vs 55.34 GB for Adam.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LDAdam instead of Adam | RoBERTa-base · GLUE Avg | score: 86.28 -> 86.48 | Helped |
| Use LDAdam instead of GaLore | RoBERTa-base · GLUE Avg | score: 84.73 -> 86.48 | Helped |
| Add generalized error feedback (ablation) | LDAdam on RoBERTa-base · GLUE Avg | score: 85.54 -> 86.48 | Helped |
| Use LDAdam r=512 instead of Adam | Llama-2 7B · GSM8K | accuracy: 34.72 -> 35.86 | Helped |
| Use LDAdam r=32 instead of Adam | Llama-2 7B · GSM8K | accuracy: 34.72 -> 32.53 | Hurt |
| Add generalized error feedback at rank 32 (ablation) | LDAdam on Llama-2 7B · GSM8K | accuracy: 30.70 -> 32.53 | Helped |
| Add generalized error feedback at rank 512 (ablation) | LDAdam on Llama-2 7B · GSM8K | accuracy: 35.78 -> 35.86 | Helped |
| Use LDAdam instead of GaLore | Llama-2 7B · GSM8K r=32 | accuracy: 27.07 -> 32.53 | Helped |
| Use LDAdam r=256 instead of Adam | Llama 130M · C4 validation | perplexity: 24.64 -> 22.65 | Helped |
| Add generalized error feedback at rank 8 (ablation) | LDAdam on Llama 130M · C4 validation | perplexity: 45.17 -> 23.82 | Helped |
| Add generalized error feedback at rank 256 (ablation) | LDAdam on Llama 130M · C4 validation | perplexity: 25.32 -> 22.65 | Helped |
| Use LDAdam r=256 instead of Adam | Llama 350M · C4 validation | perplexity: 18.08 -> 17.30 | Helped |
Surprising findings
- LDAdam without error feedback at rank 8 diverges on Llama 130M pre-training (perplexity 45.17) while LDAdam with error feedback converges (23.82).
- LDAdam slightly outperforms Adam in pre-training and fine-tuning, which the authors attribute to implicit regularization from compression.
Limitations
- Large-scale billion-parameter pre-training was not performed due to compute limits.
- Not compatible with gradient clipping or per-layer weight updates, which reduces applicability.
- Extending the projection-aware rule to other optimizers than Adam requires further work.
Experimental setup
Datasets: GLUE GSM8K C4
Benchmarks: GLUE GSM8K C4
Metrics: accuracy validation perplexity peak memory runtime
Baselines: Adam GaLore LDAdam no-EF
Builds on: Adam GaLore AMSGrad PowerSGD error feedback
Closest papers by meaning
- A Memory Efficient Randomized Subspace Optimization Method for Training Large Language Models (ICML 2025 · similarity 0.90)
- Breaking the Frozen Subspace: Importance Sampling for Low-Rank Optimization in LLM Pretraining (NeurIPS 2025 · similarity 0.89)
- On the Duality between Gradient Transformations and Adapters (ICML 2025 · similarity 0.89)
- GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection (ICML 2024 · similarity 0.88)
- APT: Adaptive Pruning and Tuning Pretrained Language Models for Efficient Training and Inference (ICML 2024 · similarity 0.88)
- Subspace Optimization for Large Language Models with Convergence Guarantees (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.