Research map / Architectures and efficiency / Pruning and efficient tuning
FM-Delta: Lossless Compression for Storing Massive Fine-tuned Foundation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FM-Delta, a lossless compression scheme that stores fine-tuned models as entropy-coded integer deltas against a single uncompressed pre-trained model, cutting storage by about 50%.
Problem
Cloud platforms storing massive numbers of full fine-tuned foundation models face heavy storage costs, and generic lossless compressors barely reduce model size.
Approach
Fine-tuned and pre-trained float parameters are mapped to unsigned integers with the same bit width, their integer difference is computed (bit-redundant because fine-tuning deltas are small), and the sign and most-significant-bit symbols are range-coded while raw bits are stored unchanged.
Key results
- FM-Delta cuts GPT-NeoX-20B storage for 10 fine-tuned models from 423.2GB to 205.2GB (48%), with about 50-60% storage reduction on average across families versus ~90% for generic compressors.
- FM-Delta reaches about 109.7 MB/s average compression throughput, the highest among the compared schemes, and 100.9 MB/s decompression.
- Theory shows model delta grows as O(T^{1/4}) with fine-tuning steps, and the most significant bit position of the integer delta rises only about 4 bits across a 65,536x value range.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FM-Delta instead of LZMA to store fine-tuned models | GPT-NeoX-20B · 10 fine-tuned models | storage after compression (GB): 298.7 -> 205.2 | Helped |
| Use FM-Delta instead of LZMA to store fine-tuned models | Falcon-40B · 10 fine-tuned models | storage after compression (GB): 621.7 -> 473.9 | Helped |
| Use FM-Delta instead of LZMA to store fine-tuned models | GPT-J-6B · 10 fine-tuned models | storage after compression (GB): 104.8 -> 73.8 | Helped |
| Use FM-Delta instead of LZMA to store fine-tuned models | GPT-2 · 100 fine-tuned models | storage after compression (GB): 43.2 -> 28.7 | Helped |
| Use FM-Delta instead of LZMA to store fine-tuned models | BERT-large-uncased · 100 fine-tuned models | storage after compression (GB): 116.1 -> 82.1 | Helped |
| Use FM-Delta instead of LZMA to store fine-tuned models | Stable Diffusion UNet · 10 fine-tuned models | storage after compression (GB): 32.5 -> 23.5 | Helped |
| Use FM-Delta instead of LZMA to store fine-tuned models | ResNet50 · 20 fine-tuned models | storage after compression (GB): 1.7 -> 1.3 | Helped |
| Use float16 instead of float32 (ablation) | FM-Delta on BERT-large-uncased · CoNLL2003 | compression rate: 68% -> 55% | Helped |
| Use bfloat16 instead of float32 (ablation) | FM-Delta on BERT-large-uncased · CoNLL2003 | compression rate: 68% -> 37% | Helped |
| Use float16 instead of float32 (ablation) | FM-Delta on BERT-large-uncased · NLI | compression rate: 63% -> 45% | Helped |
| Use bfloat16 instead of float32 (ablation) | FM-Delta on BERT-large-uncased · NLI | compression rate: 63% -> 27% | Helped |
| Increase fine-tuning steps from the start of training to 1,000 steps (ablation) | FM-Delta on GPT-2 · PTB | compression rate: 60.7% -> 64.8% | Hurt |
Surprising findings
- Generic lossless compressors (LZMA, Gzip, Zlib, BZip2) reduce fine-tuned model storage only to about 90%, showing models have little byte-level redundancy on their own.
- Compression rate improves for lower-precision types: bfloat16 compresses to about 27-37% versus 63-68% for float32 on Bert-large.
Limitations
- No GPU-side compression support; the method targets cloud storage reduction only.
- Compression rate depends on how close fine-tuned models are to the pre-trained model; large fine-tuning deltas reduce gains.
- Cost and capacity analysis relies on assumptions (binomial request model, hardware prices) and rough estimates.
Experimental setup
Datasets: Baize OpenAssistant OASST WikiText103 CoNLL2003 SST2 FER2013 Eurosat Pokemon Stable Diffusion
Benchmarks: Overall compression rate on Falcon-40B, GPT-NeoX-20B, GPT-J-6B, GPT-2, Bert-large-uncased, Stable Diffusion UNet, ResNet
Metrics: compression rate compression throughput (MB/s) decompression throughput (MB/s) end-to-end transfer time perplexity
Baselines: LZMA Gzip Zlib FPZip BZip2
Builds on: FPzip Lorenzo predictor range coding
Closest papers by meaning
- Breaking the Compression Ceiling: Data-Free Pipeline for Ultra-Efficient Delta Compression (NeurIPS 2025 · similarity 0.85)
- Delta-CoMe: Training-Free Delta-Compression with Mixed-Precision for Large Language Models (NeurIPS 2024 · similarity 0.85)
- SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression (ICLR 2024 · similarity 0.84)
- BitStack: Any-Size Compression of Large Language Models in Variable Memory Environments (ICLR 2025 · similarity 0.84)
- 70% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float (DFloat11) (NeurIPS 2025 · similarity 0.83)
- ExCP: Extreme LLM Checkpoint Compression via Weight-Momentum Joint Shrinking (ICML 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.