Research map / Optimization / Optimizers

On the Duality between Gradient Transformations and Adapters

ICML 2025 · Optimization › Optimizers · approach: Memory-efficient LLM optimizers · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that training with linear gradient transformations (e.g., GaLore) is exactly equivalent to training a linear adapter, recovering GaLore as one-sided LoRA, and uses this duality to propose quantized and distributed-initialization variants evaluated on LLaMA-style pretraining.

Problem

Memory-efficient LLM training methods (LoRA-based adapters and gradient compression like GaLore) appear distinct, and their relationship is not well understood; existing methods lack a unifying view and principled ways to combine them.

Approach

Prove that optimizing with a linear gradient transformation S is equivalent (same optimizer trajectory) to reparameterizing weights as W0 + S^T Lambda and training only Lambda; with Kronecker-factored S = R^T ⊗ L this becomes W + L^T A R^T, and GaLore is exactly one-sided LoRA. This duality is used to combine projections with QLoRA-style quantization and to design worker-aware projection initializ

Key results

• Random one-sided sketches (e.g., Gaussian, Rademacher) perform similarly to GaLore SVD projections at 200M and 1B scale; GaLore 200M perplexity 21.34 vs Gaussian 20.57 at similar memory.
• Adding 8-bit INT8 quantization to the base weights causes little perplexity loss, while NF4 degrades perplexity (e.g., GaLore 21.34 to 26.52 with NF4).
• In distributed pretraining at 200M, distributed random projection initialization (19.81 ppl) beats the distributed ReLoRA/LTE baseline (20.97) but remains behind DiLoCo (18.00).

Measured effects reported in the paper

Surprising findings

• Gradient reconstruction error does not correlate with downstream perplexity; SVD-based projections with lower reconstruction error perform similarly to random sketches.
• The paper finds a non-trivial gap between memory-efficient pretraining methods and full pretraining, contrary to the reported no-gap result in GaLore's setup.

Limitations

• Experiments limited to small scale (200M and 1B parameters) due to compute constraints.
• Weight decay in AdamW breaks the exact equivalence unless adjusted.
• Experiments focus mainly on the GaLore-LoRA special case rather than general Kronecker or other linear maps.

Experimental setup

Datasets: SlimPajama

Benchmarks: SlimPajama validation set

Metrics: perplexity estimated memory (GB) gradient reconstruction error cosine similarity

Baselines: Full pretraining ReLoRA GaLore QGaLore DiLoCo LTE (LoRA-the-Explorer) Flora

Builds on: GaLore LoRA ReLoRA QLoRA DiLoCo

Closest papers by meaning

• LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statistics (ICLR 2025 · similarity 0.89)
• AltLoRA: Towards Better Gradient Approximation in Low-Rank Adaptation with Alternating Projections (NeurIPS 2025 · similarity 0.87)
• MISA: Memory-Efficient LLMs Optimization with Module-wise Importance Sampling (NeurIPS 2025 · similarity 0.86)
• The Expressive Power of Low-Rank Adaptation (ICLR 2024 · similarity 0.86)
• LoRA-One: One-Step Full Gradient Could Suffice for Fine-Tuning Large Language Models, Provably and Efficiently (ICML 2025 · similarity 0.86)
• Memory Grafting: Scaling Language Model Pre-training via Offline Conditional Memory (NeurIPS 2026 · similarity 0.86)