Research map / Architectures and efficiency / Quantization
KBVQ-MoE: KLT-guided SVD with Bias-Corrected Vector Quantization for MoE Large Language Models
Zukang Xu, Zhixiong Zhao, Xing Hu, Zhixuan Chen, Dawei Yang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add IDRE (KLT-guided SVD shared component) (ablation) | KBVQ-MoE · WikiText2, 3-bit | perplexity: 18.72 -> 11.67 | Improved |
| add BCOS (channel-wise bias correction) (ablation) | KBVQ-MoE · WikiText2, 3-bit | perplexity: 18.72 -> 14.32 | Improved |
| add BCOS on top of IDRE (ablation) | KBVQ-MoE · WikiText2, 3-bit | perplexity: 11.67 -> 9.26 | Improved |
| add KLT guidance to SVD (ablation) | KBVQ-MoE · WikiText2, 2-bit | perplexity: 14.03 -> 11.87 | Improved |
| add SVD shared component (k/n=1/128) (ablation) | KBVQ-MoE · WikiText2, 2-bit | perplexity: 15.30 -> 11.87 | Improved |
| increase truncated rank from 1/128 to 1/32 (ablation) | KBVQ-MoE · WikiText2, 2-bit | perplexity: 11.87 -> 11.01 | No clear effect |
| add IDRE and BCOS to GPTVQ (ablation) | GPTVQ · WikiText2, 2-bit | perplexity: 12.88 -> 9.43 | Improved |
| add IDRE and BCOS to VPTQ (ablation) | VPTQ · WikiText2, 2-bit | perplexity: 10.17 -> 8.78 | Improved |
| KBVQ-MoE vs GPTQ at 3-bit on DeepseekV2-Lite | KBVQ-MoE · Avg zero-shot accuracy | accuracy: 68.89 vs 68.73 | No clear effect |
| KBVQ-MoE vs VQ at 2-bit on Qwen3-30B-A3B | KBVQ-MoE · Avg zero-shot accuracy | accuracy: 30.61 -> 63.37 | Improved |
| KBVQ-MoE vs GPTQ at 2-bit on Mixtral-8x7B | KBVQ-MoE · Avg zero-shot accuracy | accuracy: 67.56 -> 75.69 | Improved |
| quantize decoder with KBVQ-MoE at 2-bit | KBVQ-MoE · Decoder speed | speedup: 22.31 -> 35.24 tokens/s | Improved |
Closest papers by meaning
- MoE-SVD: Structured Mixture-of-Experts LLMs Compression via Singular Value Decomposition (ICML 2025 · similarity 0.92)
- MoEQuant: Enhancing Quantization for Mixture-of-Experts Large Language Models via Expert-Balanced Sampling and Affinity Guidance (ICML 2025 · similarity 0.91)
- Mixture Compressor for Mixture-of-Experts LLMs Gains More (ICLR 2025 · similarity 0.90)
- Delta Decompression for MoE-based LLMs Compression (ICML 2025 · similarity 0.88)
- FlexMoE: One-for-All Nested Intra-Expert Pruning for MoE Language Models (NeurIPS 2026 · similarity 0.88)
- QSVD: Efficient Low-rank Approximation for Unified Query-Key-Value Weight Compression in Low-Precision Vision-Language Models (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
