Research map / Architectures and efficiency / Mixture of experts
Unchosen Experts Can Contribute Too: Unleashing MoE Models’ Power by Self-Contrast
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SCMoE, a training-free decoding method that contrasts top-2 (strong) and rank-k (weak) expert activations within the same MoE model to improve reasoning and code generation.
Problem
Unchosen experts in MoE models do not contribute to the output at inference, and increasing the number of activated experts does not reliably improve quality, leaving model capacity underutilized.
Approach
Use the MoE model's own strong activation (default top-2 routing) and weak activation (rank-k routing) and contrast their next-token logits, so unchosen experts contribute to the prediction without training or extra models.
Key results
- SCMoE raises Mixtral 8x7B GSM8K accuracy from 61.79 (greedy) to 66.94.
- SCMoE improves HumanEval pass@1 from 33.54 to 41.46 and MBPP from 46.20 to 48.80 on Mixtral 8x7B.
- Combining SCMoE with self-consistency raises GSM8K major@20 from 75.59 to 78.31; SCMoE incurs about 1.30x decoding latency versus greedy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply SCMoE self-contrast decoding | Mixtral 8x7B · GSM8K | accuracy: 61.79 to 66.94 (+5.15) | Helped |
| Apply SCMoE self-contrast decoding | Mixtral 8x7B · StrategyQA | accuracy: 72.83 to 76.29 (+3.46) | Helped |
| Apply SCMoE self-contrast decoding | Mixtral 8x7B · MBPP | pass@1 accuracy: 46.20 to 48.80 (+2.60) | Helped |
| Apply SCMoE self-contrast decoding | Mixtral 8x7B · HumanEval | pass@1 accuracy: 33.54 to 41.46 (+7.92) | Helped |
| Apply SCMoE self-contrast decoding | DeepSeekMoE-16B · GSM8K | accuracy: 18.95 to 20.77 | Helped |
| Apply SCMoE self-contrast decoding | DeepSeekMoE-16B · StrategyQA | accuracy: 60.41 to 62.99 | Helped |
| Apply SCMoE self-contrast decoding | DeepSeekMoE-16B · MBPP | pass@1 accuracy: 35.20 to 37.20 | Helped |
| Apply SCMoE self-contrast decoding | DeepSeekMoE-16B · HumanEval | pass@1 accuracy: 26.83 to 28.05 | Helped |
| Replace top-2 strong activation with task-specific top-k activation (ablation) | SCMoE on Mixtral 8x7B · GSM8K | accuracy: 66.94 to 68.92 (+1.98) | Helped |
| Replace top-2 strong activation with task-specific top-k activation (ablation) | SCMoE on Mixtral 8x7B · StrategyQA | accuracy: 76.29 to 76.42 (+0.13) | Helped |
| Replace top-2 strong activation with task-specific top-k activation (ablation) | SCMoE on Mixtral 8x7B · MBPP | pass@1 accuracy: 48.80 to 50.60 (+1.80) | Helped |
| Combine SCMoE with self-consistency (ablation) | Mixtral 8x7B · GSM8K major@20 | accuracy: 75.59 to 78.31 (+2.72) | Helped |
Surprising findings
- Increasing the number of activated experts (ensemble routing) does not stably improve and can hurt performance, e.g., top-8 drops GSM8K to 57.70 from 61.79 at top-2.
- DoLa degrades Mixtral 8x7B well below greedy, often failing to terminate generation.
Limitations
- Not evaluated on larger MoE models such as Mixtral 8x22B or DeepSeek-V2 due to resource constraints.
- Gains are benchmark-dependent; the paper notes limited expected benefit on benchmarks without verbalized reasoning such as MMLU.
Experimental setup
Datasets: GSM8K StrategyQA MBPP HumanEval
Benchmarks: GSM8K StrategyQA MBPP HumanEval
Metrics: accuracy pass@1 major@20 latency ratio KL divergence
Baselines: Greedy Dynamic Routing Ensemble Routing Contrastive Search Contrastive Decoding DoLa
Builds on: contrastive decoding DoLa top-k routing self-consistency
Closest papers by meaning
- MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Experts (ICLR 2025 · similarity 0.90)
- Solving Token Gradient Conflict in Mixture-of-Experts for Large Vision-Language Model (ICLR 2025 · similarity 0.89)
- BAM! Just Like That: Simple and Efficient Parameter Upcycling for Mixture of Experts (NeurIPS 2024 · similarity 0.89)
- Domain-Specific Pruning of Large Mixture-of-Experts Models with Few-shot Demonstrations (NeurIPS 2025 · similarity 0.89)
- Advancing Expert Specialization for Better MoE (NeurIPS 2025 · similarity 0.89)
- Mixture of Lookup Experts (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.