Research map / Architectures and efficiency / Pruning and efficient tuning
Pay Attention to Small Weights
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NanoAdam, a gradient-free dynamic sparse optimizer that fine-tunes only the smallest-magnitude weights per layer, claiming better generalization, lower memory, and less catastrophic forgetting than MicroAdam, AdamW-8bit, and GaLore.
Problem
Full finetuning of large pretrained networks is memory- and compute-intensive, and existing sparse or memory-efficient optimizers either need gradient information for mask selection, keep error-feedback buffers, or cause catastrophic forgetting.
Approach
Observes a hyperbolic correlation in finetuning between large gradients and small-magnitude weights, and therefore selects the update mask purely by smallest absolute weight magnitude, refreshed dynamically with a density scheduler and no error feedback.
Key results
- On GLUE with BERT-Base, NanoAdam averages 85.65 vs 84.50 for MicroAdam and 84.58 for AdamW, with 3.58 GB memory vs 3.94 GB for AdamW.
- On continual CIFAR-10 to Flowers102 with ResNet101, NanoAdam reaches 83.77% CIFAR-10 after Task 2 vs 29.59% for AdamW, with forgetting 12.55 vs 68.14.
- With ViT-Large, NanoAdam shows the smallest parameter shift (l2 distance 0.68) and the highest average accuracy (98.95%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Dynamically update small-magnitude weights instead of using memory-efficient baseline optimizers | BERT-Base, BERT-Large, and OPT-1.3B finetuning · GLUE AVG. | average eval metric: 84.50 -> 85.65; 86.24 -> 87.48; 87.55 -> 87.92 | Helped |
| Use NanoAdam rather than AdamW-8bit or MicroAdam | GLUE finetuning · BERT-Base, BERT-Large, and OPT-1.3B | average memory usage (GB): 3.72 -> 3.58; 5.54 -> 5.18; 13.08 -> 11.60 | Helped |
| Dynamically update small-magnitude weights rather than using AdamW | ViT-Large continual finetuning · CIFAR-10 after Task 2 | accuracy (%): 98.61 -> 99.35 | Helped |
| Dynamically update small-magnitude weights rather than using AdamW | ResNet101 continual finetuning · CIFAR-10 after Task 2 and Flowers102 after Task 2 | accuracy (%): 29.59 -> 83.77; 52.97 -> 81.52 | Helped |
| Dynamically update small-magnitude weights rather than using AdamW | ResNet18 continual finetuning · CIFAR-10 after Task 2 and Flowers102 after Task 2 | accuracy (%): 27.59 -> 64.17; 56.58 -> 70.67 | Helped |
| Dynamically update small-magnitude weights rather than using AdamW | ViT-Large continual finetuning · Continual learning | Forgetting: 0.69 -> 0.02 | Helped |
| Dynamically update small-magnitude weights rather than using AdamW | ResNet101 continual finetuning · Continual learning | Forgetting: 68.14 -> 12.55 | Helped |
| Dynamically update small-magnitude weights rather than using AdamW | ResNet18 continual finetuning · Continual learning | Forgetting: 68.05 -> 28.36 | Helped |
| Select small-magnitude weights instead of large-magnitude weights (ablation) | NanoAdam on BERT-Base · SST-2 | evaluation accuracy | Helped |
| Select small-magnitude weights instead of random weights (ablation) | NanoAdam on BERT-Base · SST-2 | evaluation accuracy | Helped |
| Select small-magnitude weights instead of large-gradient parameters (ablation) | NanoAdam on BERT-Base · SST-2 | evaluation accuracy: 1 \times 10^{-3} (small weights); 3 \times 10^{-4} (large gradients) | Helped |
| Select small-magnitude weights instead of large-gradient parameters (ablation) | NanoAdam on ViT-Large · Flowers102 | evaluation accuracy: $3e-3$ | No clear effect |
Surprising findings
- Updating the largest-magnitude weights performs worse than random masking on SST-2, even though small-weight updates work best.
- The gradient-weight hyperbolic correlation is weaker when training from scratch than during finetuning.
Limitations
- Effectiveness depends on knowledge transferability between pretraining and finetuning tasks; may underperform full Adam when tasks are dissimilar.
- Benefits depend on overparameterization; smaller models such as ResNet-18 show weaker results.
- The theoretical analysis uses a two-layer teacher-student ReLU setting that is far from the transformer experiments.
Experimental setup
Datasets: GLUE CIFAR-10 Flowers102 ImageNet
Benchmarks: GLUE CIFAR-10 Flowers102
Metrics: accuracy Matthews correlation Pearson correlation peak memory (GB) forgetting l2 parameter distance
Baselines: MicroAdam AdamW-8bit GaLore AdamW
Builds on: Adam GaLore MicroAdam 8-bit Adam
Closest papers by meaning
- Upweighting Easy Samples in Fine-Tuning Mitigates Forgetting (ICML 2025 · similarity 0.87)
- PROFIT: A Specialized Optimizer for Deep Fine Tuning (NeurIPS 2025 · similarity 0.87)
- PACE: Marrying generalization in PArameter-efficient fine-tuning with Consistency rEgularization (NeurIPS 2024 · similarity 0.87)
- Inductive biases of multi-task learning and finetuning: multiple regimes of feature reuse (NeurIPS 2024 · similarity 0.86)
- LIFT the Veil for the Truth: Principal Weights Emerge after Rank Reduction for Reasoning-Focused Supervised Fine-Tuning (ICML 2025 · similarity 0.86)
- Gradient Weight-normalized Low-rank Projection for Efficient LLM Training (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.