Research map / Architectures and efficiency / Mixture of experts
Your Mixture-of-Experts LLM Is Secretly an Embedding Model for Free
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that MoE routing weights (RW) from pretrained MoE LLMs act as a training-free embedding that is complementary to hidden states, and proposes MoEE, a weighted sum of RW and HS similarities that improves MTEB performance without finetuning.
Problem
Decoder-only LLMs give weak training-free embeddings from hidden states, which are biased toward next-token prediction and sensitive to prompt choice; it is unclear whether pretrained LLMs can serve as generalist embedding models without representation finetuning.
Approach
Use the expert routing weights of MoE LLMs (concatenated across all layers, last token) as an embedding that is complementary to and more prompt-robust than hidden states, and combine them via a weighted sum of their cosine similarities (MoEE sum).
Key results
- On MTEB without prompts, MoEE (sum) on DeepSeekMoE-16B reaches average 43.30 vs 35.36 for hidden state alone (about 22% relative gain).
- With PromptEOL, MoEE (sum) reaches average 55.16 on OLMoE-1B-7B and 55.04 on Qwen1.5-MoE-A2.7B, surpassing supervised SimCSE-BERT-sup (53.53) and coCondenser-msmarco (54.55) on the reported averages.
- RW is more robust to prompt choice than HS: mean cross-prompt Spearman correlation of 0.63 for RW vs 0.52 for HS.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| combine RW and HS using weighted-sum similarity | DeepSeekMoE-16B embeddings · MTEB average, no prompt | average score: 35.36 -> 43.30 | Helped |
| combine RW and HS using weighted-sum similarity | Qwen1.5-MoE-A2.7B embeddings · MTEB average, no prompt | average score: 35.18 -> 42.25 | Helped |
| combine RW and HS using weighted-sum similarity | DeepSeekMoE-16B embeddings · MTEB average, PromptEOL | average score: 42.30 -> 53.28 | Helped |
| use last-token routing weights instead of averaging routing weights across tokens (ablation) | DeepSeekMoE-16B RW embedding · STS12-16 average | Spearman correlation: 46.03 -> 61.18 | Helped |
| use last-token hidden state from the last layer instead of mean-pooling all tokens from the last layer (ablation) | DeepSeekMoE-16B HS embedding · STS12-16 average | Spearman correlation: 32.78 -> 60.40 | Helped |
| use hidden states from all layers rather than only the last layer, for last-token extraction (ablation) | DeepSeekMoE-16B HS embedding · STS12-16 average | Spearman correlation: 60.40 -> 55.03 | Hurt |
| combine RW and HS | MoEE embeddings · MTEB tasks | performance | Helped |
| use PromptEOL prompts | MoEE · MTEB average | average score: 25.96% improvement for DeepSeekMoE | Helped |
Surprising findings
- Concatenating RW and HS embeddings underperforms a weighted sum of their similarities.
- Using the last token's routing weights outperforms averaging over all tokens, and pooling across layers introduces noise in the ablation.
Limitations
- Evaluated only on a subset of 20 MTEB datasets across three MoE models; results are not reported for dense LLMs.
- The weighted-sum hyperparameter alpha must be tuned, and the sum formulation is limited to similarity-based tasks; the paper says adapting it to other tasks is needed.
- Dataset-level gains are mixed; for some tasks such as summarization and PairCLF, MoEE does not consistently beat the best single component.
Experimental setup
Datasets: STS12 STS13 STS14 STS15 STS16 STS12 (sentence pairs for Spearman correlation analysis)
Benchmarks: MTEB
Metrics: Accuracy V-Measure Average Precision Mean Average Precision nDCG Spearman correlation
Baselines: Hidden State (HS) Routing Weight (RW) Glove Komninos BERT SimCSE-BERT-unsup SimCSE-BERT-sup coCondenser-msmarco SPECTER
Builds on: PromptEOL Mixture-of-Experts BERTopic
Closest papers by meaning
- Advancing Expert Specialization for Better MoE (NeurIPS 2025 · similarity 0.91)
- OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models (ICML 2024 · similarity 0.90)
- Pruning and Distilling Mixture-of-Experts into Dense Language Models (NeurIPS 2026 · similarity 0.90)
- Mixture Compressor for Mixture-of-Experts LLMs Gains More (ICLR 2025 · similarity 0.89)
- $\texttt{MoE-RBench}$: Towards Building Reliable Language Models with Sparse Mixture-of-Experts (ICML 2024 · similarity 0.89)
- BAM! Just Like That: Simple and Efficient Parameter Upcycling for Mixture of Experts (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.