Research map / Architectures and efficiency / Mixture of experts
RQ-MoE: Residual Quantization via Mixture of Experts for Efficient Input-Dependent Vector Compression
Zhengjia Zhong, Shuyan Ke, Zaizhou Lin, Jiaqi Song, Hongyi Lan, Hui Li
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| input-dependent two-level MoE codebooks with dual-stream instruction instead of QINCo | QINCo · BigANN1M 8 bytes, 10M training | Recall@1 (%): 45.1 -> 46.2 | Improved |
| RQ-MoE instead of QINCo | QINCo · Contriever1M 8 bytes | Recall@1 (%): 20.4 -> 21.6 | Improved |
| RQ-MoE instead of QINCo | QINCo · Deep1M 8 bytes | MSE: 0.12 -> 0.12 | No clear effect |
| RQ-MoE instead of QINCo | QINCo · FB-ssnpp1M 8 bytes | Recall@1 (%): 3.6 -> 3.6 | No clear effect |
| decouple instruction stream for parallel decoding (N=4, L=1) | QINCo · decoding latency (us) | latency: 3.3 -> 0.5 | Improved |
| use more second-level experts with fewer layers (N=4, L=1) | RQ-MoE N=1, L=4 · encoding latency (us) | latency: 96.2 -> 67.9 | Improved |
| add Normalized Residual Loss instead of per-step MSE (ablation) | RQ-MoE · Deep1M R@1, 10M training | Recall@1 (%): 30.7 -> 36.5 | Improved |
| add Normalized Residual Loss instead of per-step MSE (ablation) | RQ-MoE · BigANN1M R@1, 500K training | Recall@1 (%): 34.7 -> 42.3 | Improved |
| add expert codebook E instead of reusing base codebook (ablation) | RQ-MoE · BigANN1M R@1, 500K training | Recall@1 (%): 39.0 -> 42.3 | Improved |
| add expert codebook E instead of reusing base codebook (ablation) | RQ-MoE · FB-ssnpp1M R@1, 10M training | Recall@1 (%): 3.5 -> 3.6 | Improved |
| use NRL instead of MSE for deep quantization steps (ablation) | RQ-MoE expert codebook gradients · step 6 relative gradient | relative gradient magnitude: 0.017 -> 0.108 | Improved |
| increase expert dimensionality De beyond input dimension D (ablation) | RQ-MoE · BigANN1M | MSE | No clear effect |
Closest papers by meaning
- Residual Quantization with Implicit Neural Codebooks (ICML 2024 · similarity 0.92)
- Qinco2: Vector Compression and Search with Improved Implicit Neural Codebooks (ICLR 2025 · similarity 0.91)
- TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate (ICLR 2026 · similarity 0.88)
- KBVQ-MoE: KLT-guided SVD with Bias-Corrected Vector Quantization for MoE Large Language Models (ICLR 2026 · similarity 0.88)
- Addressing Representation Collapse in Vector Quantized Models with One Linear Layer (ICCV 2025 · similarity 0.88)
- Restructuring Vector Quantization with the Rotation Trick (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
