Research map / Architectures and efficiency / Mixture of experts
CAMERA: Multi-Matrix Joint Compression for MoE Models via Micro-Expert Redundancy Analysis
Yuzhuang Xu, Xu Han, Yuanchi Zhang, Yixuan Wang, Yijun Liu, Shiyu Ji, Qingfu Zhu, Wanxiang Che
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use CAMERA-P micro-expert pruning instead of D2-MoE/NAEE | Deepseek-MoE-16B · 9 zero-shot tasks, 60% pruned | avg accuracy: 46.72 (D2-MoE) -> 51.62 | Improved |
| use CAMERA-P instead of D2-MoE/NAEE | Qwen3-30B-A3B · 9 zero-shot tasks, 40% pruned | avg accuracy: 65.77 (NAEE) -> 67.46 | Improved |
| use CAMERA-P instead of NAEE at low pruning ratio | Deepseek-MoE-16B · 9 zero-shot tasks, 20% pruned | avg accuracy: NAEE 60.51 vs CAMERA-P 61.03 | Mixed |
| use CAMERA-Q micro-expert mixed precision instead of GPTQ | Deepseek-MoE-16B · 2.25-bit quantization avg accuracy | avg accuracy: 53.45 -> 56.56 | Improved |
| use CAMERA-Q instead of expert-level mixed precision MC | Qwen3-30B-A3B · 2.25-bit quantization avg accuracy | avg accuracy: 52.79 -> 54.75 | Improved |
| slice matrices along micro-expert (row/column consistent) instead of column-wise (ablation) | CAMERA-Q · 2.25-bit quantization Deepseek-MoE-16B | Wikitext2 perplexity: 11.28 -> 9.51 | Improved |
| slice matrices consistently per micro-expert (ablation) | CAMERA-Q · 2.25-bit quantization Deepseek-MoE-16B | avg accuracy: 52.69 -> 56.56 | Improved |
| increase calibration set size from 8 to 256 sequences (ablation) | CAMERA-P · Wikitext2 perplexity, 20% pruned Deepseek-MoE-16B | perplexity: 6.66 -> 6.56 | No clear effect |
| tune balance coefficient alpha (ablation) | CAMERA-P · Deepseek-MoE-16B 20% pruned | perplexity and avg accuracy | No clear effect |
| prune micro-experts instead of whole experts (decoding) | CAMERA-P vs D2-MoE · Deepseek-MoE-16B MoE layer, 40% pruned | decoding speedup: 1.04x -> 1.42x | Improved |
| apply CAMERA-P to dense model FFN (20%) | openPangu-7B · 9 zero-shot tasks avg | avg accuracy: 52.89 -> 51.00 | Hurt |
| add Wanda 20% unstructured pruning after CAMERA-P (ablation) | openPangu-7B · 9 zero-shot tasks avg | avg accuracy: 51.00 -> 51.11 | No clear effect |
Closest papers by meaning
- FlexMoE: One-for-All Nested Intra-Expert Pruning for MoE Language Models (NeurIPS 2026 · similarity 0.91)
- Mixture Compressor for Mixture-of-Experts LLMs Gains More (ICLR 2025 · similarity 0.91)
- MoE-SVD: Structured Mixture-of-Experts LLMs Compression via Singular Value Decomposition (ICML 2025 · similarity 0.90)
- LightMoE: Reducing Mixture-of-Experts Redundancy through Expert Replacing (NeurIPS 2026 · similarity 0.89)
- Delta Decompression for MoE-based LLMs Compression (ICML 2025 · similarity 0.89)
- Pruning and Distilling Mixture-of-Experts into Dense Language Models (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
