Research map / Architectures and efficiency / Quantization
Training Dynamics Impact Post-Training Quantization Robustness
Albert Catalan-Tatjer, Niccolò Ajroldi, Jonas Geiping
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| raise the peak learning rate during pretraining (ablation) | 160M transformer on FineWebEdu, GPTQ 3/4-bit · PTQ error vs validation loss | quantization error | Improved |
| average checkpoints along the trajectory (LAWA) instead of learning rate cooldown (ablation) | 160M transformer, constant LR, 100B tokens · 3-bit validation loss | 3-bit validation loss | Improved |
| weight averaging versus LR cooldown in full precision (ablation) | 160M transformer, constant LR · FP validation loss | validation loss | Hurt |
| apply LAWA to OLMo-1B released checkpoints | OLMo-1B · validation loss FP and 4-bit | validation loss | Improved |
| increase weight decay lambda at comparable full-precision loss (ablation) | 160M transformer, WSD, LR 3e-3 · 4-bit and 3-bit quantization error | quantization error | Improved |
| use WSD instead of cosine schedule when scaling token budget (ablation) | 70M and 160M transformers · delta PTQ vs token budget | delta_PTQ | Improved |
| vary training token budget from 10B to 100B under WSD (ablation) | 160M transformer · 4-bit quantization error after cooldown | quantization error | No clear effect |
| replace AdamW with AdamC to reduce loss-gradient norm spike (ablation) | 160M transformer, WSD · 4-bit quantization error | quantization error | No clear effect |
| model soup of OLMo2 decay runs versus individual runs (ablation) | OLMo2 family · 3-bit quantization error | quantization error | Improved |
| merge APO model with mid-training checkpoint (0.9/0.1) for final SmolLM3 | SmolLM3 · 3-bit degradation | validation loss and accuracy degradation | Improved |
| cosine versus WSD schedule at higher peak LR (ablation) | 160M transformer · 4-bit quantization error | quantization error | Mixed |
Closest papers by meaning
- Scaling Laws for Precision (ICLR 2025 · similarity 0.91)
- Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensation (AAAI 2024 · similarity 0.89)
- What Makes Quantization for Large Language Model Hard? An Empirical Study from the Lens of Perturbation (AAAI 2024 · similarity 0.89)
- Quantization Error Propagation: Revisiting Layer-Wise Post-Training Quantization (NeurIPS 2025 · similarity 0.88)
- Language models scale reliably with over-training and on downstream tasks (ICLR 2025 · similarity 0.88)
- PolyPythias: Stability and Outliers across Fifty Language Model Pre-Training Runs (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
