AI Paper Atlas

Research map / Architectures and efficiency / Mixture of experts

RQ-MoE: Residual Quantization via Mixture of Experts for Efficient Input-Dependent Vector Compression

Zhengjia Zhong, Shuyan Ke, Zaizhou Lin, Jiaqi Song, Hongyi Lan, Hui Li

ICML 2026 · Architectures and efficiency › Mixture of experts · approach: Efficient MoE language model systems · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
input-dependent two-level MoE codebooks with dual-stream instruction instead of QINCoQINCo · BigANN1M 8 bytes, 10M trainingRecall@1 (%): 45.1 -> 46.2Improved
RQ-MoE instead of QINCoQINCo · Contriever1M 8 bytesRecall@1 (%): 20.4 -> 21.6Improved
RQ-MoE instead of QINCoQINCo · Deep1M 8 bytesMSE: 0.12 -> 0.12No clear effect
RQ-MoE instead of QINCoQINCo · FB-ssnpp1M 8 bytesRecall@1 (%): 3.6 -> 3.6No clear effect
decouple instruction stream for parallel decoding (N=4, L=1)QINCo · decoding latency (us)latency: 3.3 -> 0.5Improved
use more second-level experts with fewer layers (N=4, L=1)RQ-MoE N=1, L=4 · encoding latency (us)latency: 96.2 -> 67.9Improved
add Normalized Residual Loss instead of per-step MSE (ablation)RQ-MoE · Deep1M R@1, 10M trainingRecall@1 (%): 30.7 -> 36.5Improved
add Normalized Residual Loss instead of per-step MSE (ablation)RQ-MoE · BigANN1M R@1, 500K trainingRecall@1 (%): 34.7 -> 42.3Improved
add expert codebook E instead of reusing base codebook (ablation)RQ-MoE · BigANN1M R@1, 500K trainingRecall@1 (%): 39.0 -> 42.3Improved
add expert codebook E instead of reusing base codebook (ablation)RQ-MoE · FB-ssnpp1M R@1, 10M trainingRecall@1 (%): 3.5 -> 3.6Improved
use NRL instead of MSE for deep quantization steps (ablation)RQ-MoE expert codebook gradients · step 6 relative gradientrelative gradient magnitude: 0.017 -> 0.108Improved
increase expert dimensionality De beyond input dimension D (ablation)RQ-MoE · BigANN1MMSENo clear effect

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy