Research map / Architectures and efficiency / Mixture of experts
Delta Decompression for MoE-based LLMs Compression
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Compresses MoE LLMs without retraining by splitting expert weights into a Fisher-merged shared base weight plus per-expert low-rank SVD deltas, and prunes the base with a two-phase static/dynamic column pruning.
Problem
Storage and memory footprint of MoE LLMs is prohibitive; existing expert pruning and merging lose expert diversity or degrade performance at high compression ratios.
Approach
Decompose each expert weight into a shared base weight (Fisher-weighted average of experts) plus expert-specific delta weights, compress deltas with activation-aware truncated SVD, and prune the base weight in a semi-dynamical static-then-dynamic structured manner.
Key results
- On Mixtral-8x7B at 60% compression, D2-MoE reaches 0.52 average accuracy versus 0.36 for NAEE and MoE-I^2.
- At 20% compression on Mixtral-8x7B, D2-MoE averages 0.60 vs 0.63 original, outperforming NAEE (0.58) and MoE-I^2 (0.57).
- Fisher-weighted base merging gives WikiText-2 perplexity 5.28 at 40% compression versus 6.42 for expert-frequency merging and 187.19 for RegMean.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply D²-MoE compression at 20% instead of using NAEE or MoE-I² | Mixtral-8×7B · 20% compression | average accuracy: NAEE 0.58; MoE-I² 0.57; D²-MoE 0.60 | Helped |
| Apply D²-MoE compression at 60% instead of NAEE | Mixtral-8×7B · 60% compression | average accuracy: 0.36 -> 0.52 | Helped |
| Use D²-MoE compression | DeepSeekMoE-16B-Base · 20%, 40%, and 60% compression | average accuracy: 0.54, 0.49, and 0.41 | Helped |
| Use D²-MoE compression | Phi-3.5-MoE · 40% compression | average accuracy: 0.60 | Helped |
| Use D²-MoE compression | Qwen2-57B-A14B · 40% compression | average accuracy: 0.58 | Helped |
| Use Fisher-weighted merging instead of mean averaging (ablation) | D²-MoE base weights on Mixtral-8×7B · 40% compression | WikiText-2 perplexity: 7.66 -> 5.28 | Helped |
| Use Fisher-weighted merging instead of TIES (ablation) | D²-MoE base weights on Mixtral-8×7B · 40% compression | WikiText-2 perplexity: 12.45 -> 5.28 | Helped |
| Use Fisher-weighted merging instead of RegMean (ablation) | D²-MoE base weights on Mixtral-8×7B · 40% compression | WikiText-2 perplexity: 187.19 -> 5.28 | Helped |
| Use Fisher-weighted merging instead of frequency merging (ablation) | D²-MoE base weights on Mixtral-8×7B · 40% compression | WikiText-2 perplexity: 6.42 -> 5.28 | Helped |
| Use semi-dynamic pruning instead of truncation-aware SVD for base weights (ablation) | D²-MoE on Mixtral-8×7B · base weights | WikiText-2 perplexity: 5.63 -> 5.28 | Helped |
| Use semi-dynamic pruning instead of static pruning for base weights (ablation) | D²-MoE on Mixtral-8×7B · base weights | WikiText-2 perplexity: 5.31 -> 5.28 | Helped |
| Use truncation-aware SVD instead of pruning for delta weights (ablation) | D²-MoE on Mixtral-8×7B · delta weights | WikiText-2 perplexity: 5.74 -> 5.28 | Helped |
Surprising findings
- RegMean-based base merging collapses to perplexity 187 on WikiText-2, far worse than simple averaging.
- Trimming delta weights of low-frequency experts keeps perplexity reasonable (14.71 at 75% compression with 7 trimmed experts).
Limitations
- Decomposition and pruning pipeline is complex, as acknowledged by the authors.
- Performance at 60% compression on DeepSeekMoE and Phi-3.5 degrades noticeably relative to the original model.
- Compression uses only 512 WikiText-2 calibration samples, so robustness to other calibration domains is only partly tested.
Experimental setup
Datasets: WikiText-2
Benchmarks: WikiText-2 PTB C4 OpenbookQA ARC-e WinoGrande HellaSwag ARC-c PIQA MathQA
Metrics: perplexity accuracy
Baselines: NAEE MoE-I^2 SparseGPT ASVD LoSparse MC-SMoE MoE-Compress TIES RegMean
Builds on: SVD-LLM Fisher merging Wanda delta compression (BitDelta, Delta-CoMe)
Closest papers by meaning
- Mixture Compressor for Mixture-of-Experts LLMs Gains More (ICLR 2025 · similarity 0.94)
- MoE-SVD: Structured Mixture-of-Experts LLMs Compression via Singular Value Decomposition (ICML 2025 · similarity 0.93)
- Delta-CoMe: Training-Free Delta-Compression with Mixed-Precision for Large Language Models (NeurIPS 2024 · similarity 0.90)
- FlexMoE: One-for-All Nested Intra-Expert Pruning for MoE Language Models (NeurIPS 2026 · similarity 0.89)
- LightMoE: Reducing Mixture-of-Experts Redundancy through Expert Replacing (NeurIPS 2026 · similarity 0.89)
- Pruning and Distilling Mixture-of-Experts into Dense Language Models (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.