Research map / Architectures and efficiency / Mixture of experts

LightMoE: Reducing Mixture-of-Experts Redundancy through Expert Replacing

NeurIPS 2026 · Architectures and efficiency › Mixture of experts · approach: Expert routing and specialization · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes expert replacing, a MoE compression paradigm that swaps low-importance experts for shared bases plus low-rank adapters and recovers them via annealed replacement, evaluated on OLMoE and DeepSeek-V2-Lite.

Problem

MoE LLMs have large memory footprints from storing many experts; existing expert pruning loses knowledge irreversibly and expert merging diminishes diversity and incurs high training overhead.

Approach

Replace less-important experts with a shared base (gate-weighted average of a dominant-expert group) plus expert-specific low-rank adapters, selecting experts with layer-adaptive thresholds and gradually annealing original experts into the compressed form during fine-tuning.

Key results

• At 30% compression, LightMoE averages 55.3 vs. LoRA 55.5 with 0.45B trainable parameters, matching LoRA fine-tuning.
• At 50% compression, LightMoE averages 48.1, outperforming MC-SMoE by 5.6 points and the directly replacing baseline by 3.8 points on average.
• On DeepSeek-V2-Lite at 50% compression, LightMoE averages 48.4 vs. MoBE 38.6, but LightMoE underperforms LoRA on Math (33.3 vs. 38.9 at 30%).

Surprising findings

• A simple directly-replacing baseline with LoRA already performs comparably to MC-SMoE, and at 40% exceeds it.
• Replacing extremely unimportant experts sometimes even improves downstream performance over LoRA at 30% compression.

Limitations

• Math performance on DeepSeek-V2-Lite lags LoRA, attributed to low-rank capacity limits.
• Activated MoE parameters decrease only modestly (0.81B to 0.70B) since critical experts are preserved.
• Training loss rises transiently during annealing, indicating a brief capability gap.

Experimental setup

Datasets: MetaMathQA CodeFeedback Cleaned Alpaca Dataset BDCI-21 Smart HCI NLU Challenge ChrEn

Benchmarks: GSM8K HumanEval ARC BoolQ PIQA WinoGrande BDCI-21 intent ChrEn translation

Metrics: accuracy exact match GPT-4 normalized quality score memory usage latency

Baselines: MoBE HC-SMoE MC-SMoE LoRA Full FT Replace (w/o shared) Replace (w. shared)

Builds on: LoRA MC-SMoE HC-SMoE ESFT

Closest papers by meaning

• FlexMoE: One-for-All Nested Intra-Expert Pruning for MoE Language Models (NeurIPS 2026 · similarity 0.93)
• Pruning and Distilling Mixture-of-Experts into Dense Language Models (NeurIPS 2026 · similarity 0.92)
• Mixture Compressor for Mixture-of-Experts LLMs Gains More (ICLR 2025 · similarity 0.91)
• Advancing Expert Specialization for Better MoE (NeurIPS 2025 · similarity 0.91)
• MoE-SVD: Structured Mixture-of-Experts LLMs Compression via Singular Value Decomposition (ICML 2025 · similarity 0.90)
• DiEP: Adaptive Mixture-of-Experts Compression through Differentiable Expert Pruning (NeurIPS 2025 · similarity 0.89)