Research map / Large language models / Instruction and fine-tuning
SPAM: Spike-Aware Adam with Momentum Reset for Stable LLM Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies gradient spikes in LLaMA and Pythia pre-training and proposes SPAM, an Adam variant with periodic momentum reset, spike-aware gradient clipping, and random sparse momentum for memory-efficient training.
Problem
LLM training is unstable due to gradient and loss spikes, which corrupt Adam's momentum statistics and degrade performance, and existing memory-efficient optimizers trade off against quality.
Approach
Periodically reset Adam's first and second moments to flush spike contamination, clip detected spiked gradients to sqrt(theta*v) instead of zeroing them, and use random sparse masks over moments for memory savings.
Key results
- On C4 pre-training, SPAM reaches perplexity 30.46 on LLaMA-60M vs 34.09 for Adam and 14.66 vs 16.13 on LLaMA-1B.
- Under memory-efficient settings matched to GaLore, SPAM gets 32.39 vs GaLore 34.88 on LLaMA-60M and 18.28 vs 18.95 on LLaMA-350M.
- On LLaMA2-7B commonsense fine-tuning, SPAM (d=100%) averages 66.7 vs 63.4 for full Adam fine-tuning; SPAM with d=0.25% averages 64.4 vs GaLore 63.6.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SPAM instead of Adam-based baselines | LLaMA pre-training · C4, LLaMA-60M to LLaMA-1B | validation perplexity: 30.46 vs. 34.09; 23.36 vs. 24.91; 17.42 vs. 18.77; 14.66 vs. 16.13 | Helped |
| use sparse-momentum SPAM instead of memory-matched GaLore | memory-efficient LLaMA pre-training · C4, LLaMA-60M to LLaMA-1B | validation perplexity: 32.39 vs. 34.88; 23.98 vs. 25.36; 18.28 vs. 18.95; 15.60 vs. 15.64 | Helped |
| use SPAM at d=0.25% instead of LoRA | LLaMA2-7B fine-tuning · Commonsense170K, average across 8 downstream tasks | average accuracy: 63.7±0.2 -> 64.4±0.2 | Helped |
| use full-density SPAM instead of Adam full fine-tuning | LLaMA2-7B fine-tuning · Commonsense170K, average across 8 downstream tasks | average accuracy: 63.4±0.1 -> 66.7±0.1 | Helped |
| replace GSS-based clipping with Gaussian-based clipping (ablation) | SPAM · C4 pre-training, LLaMA-60M and LLaMA-130M | perplexity: 30.46 -> 30.83; 23.36 -> 25.93 | Hurt |
| replace spike-aware clipping with gradient nullifying (ablation) | SPAM · C4 pre-training, LLaMA-60M and LLaMA-130M | perplexity: 30.46 -> 30.86; 23.36 -> 23.62 | Hurt |
| set spike threshold θ to 500 instead of 5000 (ablation) | SPAM · C4 pre-training, LLaMA-60M | perplexity: 30.46 -> 30.77 | Hurt |
| set spike threshold θ to 1000 instead of 5000 (ablation) | SPAM · C4 pre-training, LLaMA-60M | perplexity: 30.46 -> 30.59 | Helped |
| set spike threshold θ to 2500 instead of 5000 (ablation) | SPAM · C4 pre-training, LLaMA-60M | perplexity: 30.46 -> 30.57 | Helped |
| set spike threshold θ to 10000 instead of 5000 (ablation) | SPAM · C4 pre-training, LLaMA-60M | perplexity: 30.46 -> 30.82 | Hurt |
| select sparse parameter subsets randomly instead of by weight or gradient magnitude (ablation) | SPAM sparse momentum · C4, LLaMA-60M | training performance | Helped |
| reduce momentum reset interval from 2500 to 500 steps (ablation) | SPAM · C4, LLaMA-130M | training performance | Helped |
Surprising findings
- Simply zeroing out detected gradient spikes improves perplexity over Adam, indicating spikes are harmful rather than informative.
- Gradient spikes of up to 1000x typical magnitude coincide with small loss bumps, and spike-mitigation architectures such as Scaled Initialization and Embed Detach barely reduce them.
Limitations
- Theoretical analysis is preliminary and relies on convexity and bounded-gradient assumptions that do not match LLM training.
- Momentum reset requires hyperparameters (reset interval, warmup steps, threshold theta) tuned per setting, and too-frequent resets degrade performance.
- Sparse momentum with d=0.25% on fine-tuning requires near-full-memory cost (36G) comparable to GaLore, and gains over full-rank SPAM are not free at full density.
Experimental setup
Datasets: C4 SlimPajama Commonsense170K ImageNet-1K Weather time series
Benchmarks: C4 validation perplexity BoolQ PIQA SIQA HellaSwag WinoGrande ARC-e ARC-c OBQA ImageNet-1K
Metrics: validation perplexity accuracy memory (GB) time per iteration
Baselines: Adam Adam+Gradient-Clip-Value Adam+Gradient-Clip-Norm Adafactor Adam-mini GaLore LoRA ReLoRA Embed LN Embed Detach Scaled Embed Scaled Initialization
Builds on: Adam Adafactor GaLore Adam-mini
Closest papers by meaning
- In Search of Adam’s Secret Sauce (NeurIPS 2025 · similarity 0.89)
- Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation is Wasteful (NeurIPS 2025 · similarity 0.88)
- Mini-batch Coresets for Memory-efficient Language Model Training on Data Mixtures (ICLR 2025 · similarity 0.88)
- AdaRankGrad: Adaptive Gradient Rank and Moments for Memory-Efficient LLMs Training and Fine-Tuning (ICLR 2025 · similarity 0.88)
- Polar Sparsity: High Throughput Batched LLM Inferencing with Scalable Contextual Sparsity (NeurIPS 2025 · similarity 0.87)
- Benchmarking Optimizers for Large Language Model Pretraining (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.