Research map / Architectures and efficiency / Mixture of experts
MoE-SVD: Structured Mixture-of-Experts LLMs Compression via Singular Value Decomposition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MoE-SVD, a training-free SVD-based compression for MoE LLMs that selectively decomposes expert layers, shares a V-matrix across experts, and trims U-matrices, achieving ~20-60% compression with 1.2-1.5x inference speedup.
Problem
SVD-based LLM compressors collapse when applied directly to MoE LLMs (e.g., Mixtral-8x7B perplexity above 1000 with ASVD and SVD-LLM), and existing MoE compressors either need retraining or yield little speedup.
Approach
Layer-wise selective SVD decomposition guided by a sensitivity metric (expert sampling frequency × principal rank × activation outlier ratio), followed by sharing one frequently-routed V-matrix across all experts and keeping only the top-2 frequently-routed U-matrices per expert.
Key results
- On Mixtral-8x7B at 20% compression, MoE-SVD reaches 4.86 WikiText-2 perplexity and 0.58 average accuracy (original 0.63), versus 9.44 and 0.51 for ASVD.
- At 60% compression on Mixtral-8x7B, MoE-SVD achieves 1.8x runtime throughput gain with 13.52 WikiText-2 perplexity, while ASVD and SVD-LLM exceed 10,000 perplexity.
- LoRA fine-tuning (MoE-SVD†) raises Mixtral-8x7B 20% average accuracy from 0.58 to 0.60.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply MoE-SVD compression at 20% instead of using Wanda pruning | Mixtral-8×7B · 20% compression, WikiText-2 | perplexity: 4.72 -> 4.86 | Mixed |
| Apply MoE-SVD compression at 20% instead of using Wanda pruning | Mixtral-8×7B · 20% compression, average accuracy | accuracy: 0.57 -> 0.58 | Helped |
| Apply MoE-SVD compression at 20% | Phi-3.5-MoE · 20% compression, WikiText-2 | perplexity: 7.22 -> 4.58 | Helped |
| Apply MoE-SVD compression at 20% | Phi-3.5-MoE · 20% compression, average accuracy | accuracy: 0.56 -> 0.60 | Helped |
| Add LoRA fine-tuning to MoE-SVD (ablation) | Mixtral-8×7B · 20% compression, average accuracy | accuracy: 0.58 -> 0.60 | Helped |
| Add LoRA fine-tuning to MoE-SVD (ablation) | Phi-3.5-MoE · 20% compression, average accuracy | accuracy: 0.60 -> 0.61 | Helped |
| Use selective decomposition rather than uniform SVD (ablation) | Mixtral-8×7B · 20% compression, WikiText-2 | perplexity: 18.80 -> 8.67 | Helped |
| Use selective decomposition rather than OWL-based SVD (ablation) | Mixtral-8×7B · 20% compression, WikiText-2 | perplexity: 16.57 -> 8.67 | Helped |
| Add matrix sharing and trimming to selective decomposition (ablation) | Mixtral-8×7B · 20% compression, WikiText-2 | perplexity: 8.67 -> 4.86 | Helped |
| Use the combined sensitivity metric fi · pi · ai rather than individual or pairwise metrics (ablation) | MoE-SVD on Mixtral-8×7B · WikiText-2 | perplexity: 12.65 -> 8.67 | Helped |
| Retain 2 U-matrices instead of retaining all 8 (ablation) | MoE-SVD on Mixtral-8×7B · WikiText-2 | perplexity: 4.86 -> 5.98 | Hurt |
| Use C4 rather than WikiText-2 for calibration (ablation) | MoE-SVD on Mixtral-8×7B · WikiText-2 | perplexity: 4.86 -> 4.91 | No clear effect |
Surprising findings
- Directly applying SVD-based methods (ASVD, SVD-LLM) to MoE LLMs collapses performance, with perplexities above 1000 on Mixtral-8x7B at 50% compression.
- OWL activation-outlier layer importance does not match the per-layer decomposition sensitivity observed in MoE models.
Limitations
- SVD-based methods add communication overhead under tensor parallelism; authors did not validate this empirically.
- Compression pipeline requires sequential per-layer calibration and SVD, which is time-consuming (about 44 minutes for 20% on Phi-3.5-MoE).
Experimental setup
Datasets: WikiText-2
Benchmarks: WikiText-2 PTB C4 OpenbookQA WinoGrande HellaSwag PIQA MathQA ARC-e ARC-c
Metrics: perplexity accuracy throughput (tokens/sec) memory usage
Baselines: Wanda SparseGPT LoSparse ASVD SVD-LLM MoE-I2 MC-SMoE MoE-Compression
Builds on: ASVD SVD-LLM OWL
Closest papers by meaning
- Delta Decompression for MoE-based LLMs Compression (ICML 2025 · similarity 0.93)
- Mixture Compressor for Mixture-of-Experts LLMs Gains More (ICLR 2025 · similarity 0.93)
- Pruning and Distilling Mixture-of-Experts into Dense Language Models (NeurIPS 2026 · similarity 0.90)
- LightMoE: Reducing Mixture-of-Experts Redundancy through Expert Replacing (NeurIPS 2026 · similarity 0.90)
- FlexMoE: One-for-All Nested Intra-Expert Pruning for MoE Language Models (NeurIPS 2026 · similarity 0.90)
- MoEQuant: Enhancing Quantization for Mixture-of-Experts Large Language Models via Expert-Balanced Sampling and Affinity Guidance (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.