Research map / Architectures and efficiency / Pruning and efficient tuning
A Better Start: Sensitivity-Aware Warm-Up for Robust and Efficient Fine-Tuning
Yile Chen, Zeyi Wen, Jian Chen, Jin Huang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add SAWU to Adam fine-tuning | Adam + linear warm-up · RoBERTa-base GLUE (6 tasks) | test accuracy: e.g. QNLI 90.24 -> 92.62; CoLA 55.75 -> 63.12 | Improved |
| add SAWU to Adam fine-tuning | Adam · MRPC (RoBERTa-base) | test accuracy: 85.86 -> 87.59 | Improved |
| add SAWU to RAdam | RAdam · QNLI (RoBERTa-base) | test accuracy: 49.46 -> 91.78 | Improved |
| add SAWU to LAMB | LAMB · MNLI (RoBERTa-base) | test accuracy: 86.49 -> 86.37 | Hurt |
| add SAWU to AdamW | AdamW · MNLI (RoBERTa-base) | test accuracy: 83.85 -> 86.43 | Improved |
| add SAWU to LLaMA LoRA fine-tuning | Adam · LLaMA-3.2-1B LoRA, avg of 4 GLUE tasks | accuracy: 89.72 -> 91.10 | Improved |
| add sensitivity-aware loss (ablation) | vanilla warm-up Adam · MRPC (RoBERTa-base) | warm-up phase accuracy: 76.06 -> 83.01 | Improved |
| add sensitivity-aware loss (ablation) | vanilla warm-up Adam · MRPC (RoBERTa-base) | decay phase accuracy: 85.86 -> 87.51 | Improved |
| add sensitivity-guided scheduling on top of loss (ablation) | SAWU (+SL) · MRPC (RoBERTa-base) | warm-up phase accuracy: 83.01 -> 82.96 | No clear effect |
| add adaptive phase transition (ablation) | SAWU (+SL+SS) · MRPC (RoBERTa-base) | training time (decay phase, s): 61 -> 60 | Improved |
| increase sliding window w to 9 (ablation) | SAWU · MRPC (RoBERTa-base) | warm-up accuracy | Hurt |
| set max count k to 3 instead of 7-9 (ablation) | SAWU · MRPC (RoBERTa-base) | warm-up accuracy: 67.99 for k=3 | Hurt |
Closest papers by meaning
- Where Does Warm-Up Come From? Adaptive Scheduling for Norm-Constrained Optimizers (NeurIPS 2026 · similarity 0.89)
- Sheared Backpropagation for Fine-tuning Foundation Models (CVPR 2024 · similarity 0.86)
- Analyzing & Reducing the Need for Learning Rate Warmup in GPT Training (NeurIPS 2024 · similarity 0.86)
- Cost-Sensitive Freeze-thaw Bayesian Optimization for Efficient Hyperparameter Tuning (NeurIPS 2025 · similarity 0.86)
- RoSA: Rotational Sparse Adaptation for Memory-Efficient Fine-Tuning (NeurIPS 2026 · similarity 0.86)
- MGUP: A Momentum-Gradient Alignment Update Policy for Stochastic Optimization (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
