Research map / Architectures and efficiency / Quantization
ExCP: Extreme LLM Checkpoint Compression via Weight-Momentum Joint Shrinking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Compresses LLM training checkpoints by pruning residuals of adjacent weight checkpoints jointly with Adam momentum, then non-uniformly quantizing and 7zip-compressing, reaching ~70x on Pythia-410M with near-lossless downstream accuracy.
Problem
Storing frequent training checkpoints (weights plus Adam optimizer momentum) for LLMs is extremely storage-expensive, and prior checkpoint compressors largely ignore optimizer momentum and lose accuracy on LLMs.
Approach
Compute residuals between adjacent weight checkpoints (sparser than raw weights), jointly prune residual weights and Adam momentum using optimizer statistics as indicators (weight-momentum joint shrinking), then apply K-means non-uniform quantization and 7zip; the authors also give a convergence analysis for the momentum pruning.
Key results
- On Pythia-410M, ExCP reduces checkpoint size from 4.53G to 0.06G (~70x) with average accuracy 42.93 vs 43.11 for the original model.
- On ViT-L32/ImageNet-1K, ExCP with both M2W and W2M pruning reaches 35.21x compression over weights and momentum at 71.69% top-1 vs 71.36% baseline.
- On PanGu-π-7B, storage drops from 98.59G to 4.10G (~25x) with average accuracy 57.09 vs 56.71 original.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add ExCP checkpoint compression | ViT-L32 · ImageNet-1K | CR (Weights & Momentum): 1 -> 19.88 | Helped |
| Add ExCP checkpoint compression | Pythia-410M · Pythia-410M downstream tasks | Avg: 43.11 -> 42.93 | No clear effect |
| Add ExCP checkpoint compression | PanGu-π-1B · PanGu-π-1B benchmarks | Avg: 40.74 -> 40.83 | No clear effect |
| Add ExCP checkpoint compression | PanGu-π-7B · PanGu-π-7B benchmarks | Avg: 56.71 -> 57.09 | Helped |
| Add residual calculation (ablation) | Pythia-410M checkpoint compression · Pythia-410M | Size: 4070M -> 3484M | Helped |
| Add pruning without residual calculation or quantization (ablation) | Pythia-410M · Pythia-410M downstream tasks | Avg Acc: 43.11 -> 29.95 | Hurt |
| Add quantization without residual calculation or pruning (ablation) | Pythia-410M checkpoint compression · Pythia-410M | Size: 4070M -> 492M | Helped |
| Add pruning to residual calculation (ablation) | Pythia-410M checkpoint compression · Pythia-410M | Size: 3484M -> 276M | Helped |
| Add quantization to residual calculation (ablation) | Pythia-410M checkpoint compression · Pythia-410M | Size: 3484M -> 493M | Helped |
| Add quantization to residual calculation and pruning (ablation) | Pythia-410M checkpoint compression · Pythia-410M | Size: 276M -> 61M | Helped |
| Use 2-bit instead of 4-bit quantization (ablation) | Pythia-410M · Pythia-410M downstream tasks | Avg Acc: 42.93 -> 42.46 | Hurt |
| Use 8-bit instead of 4-bit quantization (ablation) | Pythia-410M · Pythia-410M downstream tasks | Avg Acc: 42.93 -> 42.90 | No clear effect |
Surprising findings
- Pruning directly on raw weights harms accuracy substantially (average 29.95 on Pythia-410M), whereas pruning residuals does not.
- Compressed PanGu-π-1B slightly outperforms the original on some benchmarks and the authors attribute this to regularization effects.
Limitations
- Evaluation limited to a few models (ViT-L32, Pythia-410M, PanGu-π 1B/7B) and downstream tasks; no evaluation at the largest scales.
- Convergence analysis is described by the authors as a proof-of-concept and relies on assumptions from Adam's analysis that are known to be problematic.
- Non-uniform quantization does not accelerate inference, and PanGu-π compression is lower since weights are stored in float16.
Experimental setup
Datasets: ImageNet-1K The Pile
Benchmarks: ImageNet-1K HellaSwag ARC-easy PIQA C3 CSL LAMBADA C-Eval CMMLU MMLU AGI-Eval BoolQ
Metrics: top-1 accuracy average accuracy compression ratio checkpoint size
Baselines: CNR+ QD+ DynaQuant Residual+7Zip original checkpoint
Builds on: Adam LC-Checkpoint Check-N-Run QD-Compressor DynaQuant
Closest papers by meaning
- SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression (ICLR 2024 · similarity 0.88)
- Compressing LLMs: The Truth is Rarely Pure and Never Simple (ICLR 2024 · similarity 0.88)
- BitStack: Any-Size Compression of Large Language Models in Variable Memory Environments (ICLR 2025 · similarity 0.88)
- LLMCBench: Benchmarking Large Language Model Compression for Efficient Deployment (NeurIPS 2024 · similarity 0.87)
- BiLLM: Pushing the Limit of Post-Training Quantization for LLMs (ICML 2024 · similarity 0.87)
- Mixture Compressor for Mixture-of-Experts LLMs Gains More (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.