Research map / Large language models / Instruction and fine-tuning
Delta-CoMe: Training-Free Delta-Compression with Mixed-Precision for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Delta-CoMe, a training-free delta compression method that applies mixed-precision quantization to SVD singular vectors of fine-tuned-minus-backbone weights, allocating higher bits to larger singular values, and matches full fine-tuned LLMs at a 1/16 compression ratio.
Problem
Compressing the delta weights between a fine-tuned LLM and its backbone to serve many models cheaply, since existing low-rank and 1-bit (BitDelta) methods substantially degrade task-specific models such as math and code LLMs.
Approach
Exploit the long-tail distribution of singular values of delta weights by quantizing singular vectors with mixed bit-widths (8/3/2 bits for successive groups) and dropping the smallest ones (0-bit), giving a hybrid of low-rank and low-bit compression at the same size as BitDelta.
Key results
- On 7B aligned models, Delta-CoMe averages 53.2 versus 53.5 for full aligned models, compared with 47.8 for low-rank and 49.3 for BitDelta at 1/16 compression.
- Delta-CoMe reaches 41.9 average versus 29.8 for LoRA on math and code tasks, close to the aligned model's 43.0.
- The Triton kernel gives roughly 3x speedup over the PyTorch implementation, and deploying 50 models on one 80G GPU is possible with delta compression versus OOM beyond 4 models without it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace low-rank compression with Delta-CoMe at 1/16 | 7B aligned models · eight-task average | average score: 47.8 -> 53.2 | Helped |
| replace BitDelta with Delta-CoMe at 1/16 | 7B aligned models · eight-task average | average score: 49.3 -> 53.2 | Helped |
| replace low-rank compression with Delta-CoMe at 1/16 | 13B aligned models · eight-task average | average score: 51.0 -> 53.5 | Helped |
| replace BitDelta with Delta-CoMe at 1/16 | 13B aligned models · eight-task average | average score: 50.9 -> 53.5 | Helped |
| replace LoRA with Delta-CoMe | math and code models · four-benchmark average | average score: 29.8 -> 41.9 | Helped |
| add 8-bit and 2-bit vector groups to the single-precision 3-bit setting (ablation) | Delta-CoMe · WizardMath-7B GSM8K | accuracy: 51.8 -> 53.6 | Helped |
| use genetic instead of greedy bit allocation (ablation) | Delta-CoMe · 7B aligned models, eight-task average | average score: 53.2 -> 53.7 | Helped |
| increase compression from 1/16 to 1/32 (ablation) | Delta-CoMe on WizardMath-7B · GSM8K | accuracy: 53.6 -> 48.8 | Hurt |
| use Delta-CoMe instead of full fine-tuning | 7B aligned models · eight-task average | average score: 53.5 -> 53.2 | Hurt |
Surprising findings
- BitDelta, though near-lossless on chat models, significantly degrades math and code LLMs.
- Genetic search over bit allocation (using C4 perplexity) outperforms greedy search, especially on code, and even surpasses the half-precision aligned models on average (53.7 vs 53.5).
Limitations
- The Triton kernel is simple and not as optimized as existing advanced kernels, per the authors.
- Mixed-precision bit allocation is chosen by a preliminary greedy search and is not extensively explored beyond the chosen configuration.
- Performance declines noticeably at 1/32 compression ratio.
Experimental setup
Datasets: C4 GSM8K MATH HumanEval MBPP TruthfulQA SafetyBench GQA TextVQA
Benchmarks: GSM8K MATH HumanEval MBPP SafetyBench TruthfulQA GQA TextVQA
Metrics: accuracy pass@1 average score activation approximation error inference time GPU memory cost
Baselines: Low-Rank SVD compression BitDelta LoRA
Builds on: BitDelta GPTQ low-rank delta compression
Closest papers by meaning
- Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMs (ICML 2024 · similarity 0.91)
- BiLLM: Pushing the Limit of Post-Training Quantization for LLMs (ICML 2024 · similarity 0.91)
- PV-Tuning: Beyond Straight-Through Estimation for Extreme LLM Compression (NeurIPS 2024 · similarity 0.91)
- Norm Tweaking: High-Performance Low-Bit Quantization of Large Language Models (AAAI 2024 · similarity 0.91)
- SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Models (ICML 2025 · similarity 0.91)
- Compressing Large Language Models by Joint Sparsification and Quantization (ICML 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.