Research map / Large language models / Retrieval and knowledge
XPERT: Expert Knowledge Transfer for Effective Training of Language Models
Chang Liu, boyu shi, Xu Yang, Xin Geng
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Initialize FFN layers from consolidated MoE expert knowledge (XPERT-OLMoE) instead of random init | dense Llama-style 570M · language understanding SFT average | average accuracy (%): 51.81 -> 54.10 | Improved |
| Use XPERT initialization instead of distillation from OLMoE-7B | dense Llama-style 570M · dialogue generation average Rouge-L | Rouge-L: 16.54 -> 18.50 | Improved |
| Use XPERT initialization instead of scratch | dense Llama-style 570M · S-NI generation | Rouge-L: +4.21 over Scratch, +3.76 over Distillation | Improved |
| Use XPERT-OLMoE initialization | dense Llama-style 570M · DollyEval fine-tuning | convergence speed: 2.4x earlier to same Rouge-L; 1.7x faster loss | Improved |
| Use XPERT-OLMoE init with 2B pre-training tokens instead of Scratch with 10B tokens | dense Llama-style 570M · UnNI | Rouge-L | Improved |
| Select only the top-1 expert instead of multiple cross-domain experts (ablation) | XPERT · BoolQ/Hellaswag/PIQA/WinoG average | average accuracy (%): 51.92 -> 49.83 | Hurt |
| Select experts by cross-domain activation instead of randomly (ablation) | XPERT · BoolQ/Hellaswag/PIQA/WinoG average | average accuracy (%): 50.16 -> 51.92 | Improved |
| Use Tucker decomposition instead of average aggregation (ablation) | XPERT · BoolQ/Hellaswag/PIQA/WinoG average | average accuracy (%): 49.60 -> 51.92 | Improved |
| Use Tucker decomposition instead of CP decomposition (ablation) | XPERT · BoolQ/Hellaswag/PIQA/WinoG average | average accuracy (%): 49.45 -> 51.92 | Improved |
| Use Tucker decomposition instead of SVD (ablation) | XPERT · BoolQ/Hellaswag/PIQA/WinoG average | average accuracy (%): 49.68 -> 51.92 | Improved |
| Use structure-preserving parameter-scale adaptation instead of truncation (ablation) | XPERT · BoolQ/Hellaswag/PIQA/WinoG average | average accuracy (%): 49.38 -> 51.92 | Improved |
| Select experts using mixed-domain data instead of a single domain (ablation) | XPERT · downstream SFT | downstream performance | Improved |
Closest papers by meaning
- Advancing Expert Specialization for Better MoE (NeurIPS 2025 · similarity 0.90)
- FlexMoE: One-for-All Nested Intra-Expert Pruning for MoE Language Models (NeurIPS 2026 · similarity 0.89)
- From Experts to Sub-experts: Fine-grained Parameter-Efficient Fine-Tuning for MoE LLMs (NeurIPS 2026 · similarity 0.89)
- BAM! Just Like That: Simple and Efficient Parameter Upcycling for Mixture of Experts (NeurIPS 2024 · similarity 0.89)
- Your Mixture-of-Experts LLM Is Secretly an Embedding Model for Free (ICLR 2025 · similarity 0.89)
- MoE-LPR: Multilingual Extension of Large Language Models Through Mixture-of-Experts with Language Priors Routing (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
