Research map / Architectures and efficiency / Mixture of experts
TileQ: Efficient Low-Rank Quantization of Mixture-of-Experts with 2D Tiling
Hongyaoxing Gu, Xinzhe Chen, Lijuan Hu, Liu fangfang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add low-rank component (ablation) | TileQ-v (Qwen1.5-MoE-A2.7B) · WikiText-2 2-bit | perplexity: 12.5 -> 7.35 | Improved |
| add low-rank component (ablation) | TileQ-v (Qwen1.5-MoE-A2.7B) · WikiText-2 3-bit | perplexity: 7.98 -> 6.93 | Improved |
| use vector quantization instead of scalar (ablation) | TileQ (Qwen1.5-MoE-A2.7B) · WikiText-2 2-bit | perplexity: 7.56 -> 7.35 | Improved |
| use vector quantization instead of scalar (ablation) | TileQ (Qwen1.5-MoE-A2.7B) · WikiText-2 3-bit | perplexity: 6.94 -> 6.93 | No clear effect |
| add rotation (ablation) | TileQ-v (Qwen1.5-MoE-A2.7B) · WikiText-2 2-bit | perplexity: 7.6 -> 7.35 | Improved |
| use 2D-tiling instead of per-expert low-rank (ablation) | TileQ-v (Qwen1.5-MoE-A2.7B) · WikiText-2 2-bit | perplexity: 7.49 -> 7.35 | No clear effect |
| use 2D-tiling instead of per-expert low-rank (ablation) | TileQ-v · extra bits per weight | extra bit-width: 0.31 -> 0.04 | Improved |
| use TileQ-v instead of GPTQ | Mixtral-8x7B · WikiText-2 2-bit | perplexity: 15.3 -> 4.78 | Improved |
| use TileQ-s instead of LoPRo | Mixtral-8x7B · WikiText-2 3-bit | perplexity: 4.13 -> 4.10 | No clear effect |
| use vector instead of scalar TileQ at 3-bit | TileQ · 3-bit quantization | accuracy | No clear effect |
| use fused LoTileMoE 2D-tiling low-rank inference instead of element-wise or 1D MiLo | TileQ · MoE MLP block latency prefill | latency: less than 5% | Improved |
| use 1D-shared (MiLo) or element-wise low-rank at decode | MiLo-1D, element-wise · Qwen3-MoE decode | latency: around 75% latency | Hurt |
Closest papers by meaning
- Mixture Compressor for Mixture-of-Experts LLMs Gains More (ICLR 2025 · similarity 0.89)
- MoEQuant: Enhancing Quantization for Mixture-of-Experts Large Language Models via Expert-Balanced Sampling and Affinity Guidance (ICML 2025 · similarity 0.89)
- Quant Experts: Token-aware Adaptive Error Reconstruction with Mixture of Experts for Large Vision-Language Models Quantization (CVPR 2026 · similarity 0.88)
- Toward Efficient Inference for Mixture of Experts (NeurIPS 2024 · similarity 0.88)
- A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Experts (ICML 2024 · similarity 0.88)
- Pruning and Distilling Mixture-of-Experts into Dense Language Models (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
