Research map / Architectures and efficiency / Pruning and efficient tuning
Breaking the Compression Ceiling: Data-Free Pipeline for Ultra-Efficient Delta Compression
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UltraDelta, a data-free delta compression pipeline combining variance-based layer sparsity allocation, quantize-then-group-wise random pruning, and trace-norm-guided rescaling to reach up to 50x (LLMs), 224x (T5-base), and 132x (ViT-L/14) compression while matching or beating fine-tuned models.
Problem
Delta compression of fine-tuned models must keep high compression ratios and performance without relying on calibration or validation data; existing pruning and quantization methods degrade sharply at ultra-high compression.
Approach
Three data-free components operating at different granularities: variance-based inter-layer sparsity allocation (lower sparsity for high-variance layers), uniform quantization followed by value-grouped random pruning to preserve intra-layer distribution shape, and a global rescaling factor gamma/(1-s) with gamma set inversely to the delta weight's trace norm.
Key results
- On LLaMA-2 at 50.9x compression for 13B, UltraDelta averages 52.05 vs 50.94 for the fine-tuned models.
- On T5-base at 224.6x compression, averages 86.74 vs 86.37 for fine-tuned models on GLUE.
- On ViT-L/14 at 132.5x compression, averages 94.4, matching the individual fine-tuned models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Compress LLaMA-2 7B deltas with UltraDelta at 32.9× rather than storing the individual fine-tuned models | LLaMA-2 7B · 3 tasks | average score: 45.37 -> 45.57 | Helped |
| Compress LLaMA-2 13B deltas with UltraDelta at 50.9× rather than storing the individual fine-tuned models | LLaMA-2 13B · 3 tasks | average score: 50.94 -> 52.05 | Helped |
| Use UltraDelta compression on recent LLMs | Recent LLMs · 6 tasks | average score: 51.69 -> 52.05 | Helped |
| Compress T5-base deltas with UltraDelta at 224.6× | T5-base · GLUE 8 tasks | average score: 86.37 -> 86.74 | Helped |
| Compress ViT-L/14 deltas with UltraDelta at 132.5× | ViT-L/14 · 8 image classification tasks | average accuracy: 94.4 -> 94.4 | No clear effect |
| Compress BEiT-3 deltas with UltraDelta | BEiT-3 · COCO Captioning | CIDEr: 1.250 -> 1.270 | Helped |
| Add group-wise pruning after quantization to DARE (ablation) | ViT-B/32 · 8 image classification tasks, 97% sparsity | average accuracy: 89.7 -> 89.7 | No clear effect |
| Add variance-based sparsity allocation (ablation) | DARE + group-wise pruning · ViT-B/32, 8 image classification tasks, 97% sparsity | average accuracy: +0.6 | Helped |
| Add trace-norm-guided rescaling (ablation) | DARE + group-wise pruning + variance-based allocation · ViT-B/32, 8 image classification tasks, 97% sparsity | average accuracy: +0.4 | Helped |
| Use group-wise pruning with 4-bit quantization instead of DARE (ablation) | ViT-B/16 · 30 image classification tasks, 90% sparsity | average accuracy: 81.58 -> 86.40 | Helped |
| Increase quantization from 4-bit to 8-bit with group-wise pruning (ablation) | ViT-B/16 · 30 image classification tasks, 90% sparsity | average accuracy: 86.40 -> 90.18 | Helped |
| Remove quantization and use interval-based grouping (ablation) | ViT-B/16 · 30 image classification tasks, 90% sparsity | average accuracy: 86.40 -> 92.45 | Helped |
Surprising findings
- Compressed models sometimes exceed the average performance of the fine-tuned models, which the authors attribute to a regularization effect on underfitted models.
Limitations
- The rescaling strategy is heuristic rather than derived.
- Does not explicitly target deployment efficiency.
- Compression ratios are reported under Golomb coding assumptions, with index-free ideal ratios much higher and practical CSR/BCSR ratios lower.
Experimental setup
Datasets: GSM8K HumanEval TruthfulQA GLUE SUN397 Stanford Cars RESISC45 EuroSAT SVHN GTSRB MNIST DTD
Benchmarks: GSM8K HumanEval TruthfulQA HumanEval+ MBPP+ MATH GPQA MMLU BBQ GLUE SUN397 Stanford Cars
Metrics: accuracy pass@1 BLEU4 CIDEr METEOR ROUGE-L compression ratio Spearman correlation Matthews correlation
Baselines: BitDelta Delta-CoMe DARE Magnitude Pruning Multi-task Learning
Builds on: DARE BitDelta Delta-CoMe Magnitude Pruning
Closest papers by meaning
- Delta-CoMe: Training-Free Delta-Compression with Mixed-Precision for Large Language Models (NeurIPS 2024 · similarity 0.89)
- BitStack: Any-Size Compression of Large Language Models in Variable Memory Environments (ICLR 2025 · similarity 0.89)
- EvoPress: Accurate Dynamic Model Compression via Evolutionary Search (ICML 2025 · similarity 0.89)
- Compressing LLMs: The Truth is Rarely Pure and Never Simple (ICLR 2024 · similarity 0.89)
- SoLA: Leveraging Soft Activation Sparsity and Low-Rank Decomposition for Large Language Model Compression (AAAI 2025 · similarity 0.89)
- LLMC+: Benchmarking Vision-Language Model Compression with a plug-and-play Toolkit (AAAI 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.