AI Paper Atlas

Research map / Large language models / Retrieval and knowledge

XPERT: Expert Knowledge Transfer for Effective Training of Language Models

Chang Liu, boyu shi, Xu Yang, Xin Geng

ICML 2026 · Large language models › Retrieval and knowledge · approach: Knowledge editing in LLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Initialize FFN layers from consolidated MoE expert knowledge (XPERT-OLMoE) instead of random initdense Llama-style 570M · language understanding SFT averageaverage accuracy (%): 51.81 -> 54.10Improved
Use XPERT initialization instead of distillation from OLMoE-7Bdense Llama-style 570M · dialogue generation average Rouge-LRouge-L: 16.54 -> 18.50Improved
Use XPERT initialization instead of scratchdense Llama-style 570M · S-NI generationRouge-L: +4.21 over Scratch, +3.76 over DistillationImproved
Use XPERT-OLMoE initializationdense Llama-style 570M · DollyEval fine-tuningconvergence speed: 2.4x earlier to same Rouge-L; 1.7x faster lossImproved
Use XPERT-OLMoE init with 2B pre-training tokens instead of Scratch with 10B tokensdense Llama-style 570M · UnNIRouge-LImproved
Select only the top-1 expert instead of multiple cross-domain experts (ablation)XPERT · BoolQ/Hellaswag/PIQA/WinoG averageaverage accuracy (%): 51.92 -> 49.83Hurt
Select experts by cross-domain activation instead of randomly (ablation)XPERT · BoolQ/Hellaswag/PIQA/WinoG averageaverage accuracy (%): 50.16 -> 51.92Improved
Use Tucker decomposition instead of average aggregation (ablation)XPERT · BoolQ/Hellaswag/PIQA/WinoG averageaverage accuracy (%): 49.60 -> 51.92Improved
Use Tucker decomposition instead of CP decomposition (ablation)XPERT · BoolQ/Hellaswag/PIQA/WinoG averageaverage accuracy (%): 49.45 -> 51.92Improved
Use Tucker decomposition instead of SVD (ablation)XPERT · BoolQ/Hellaswag/PIQA/WinoG averageaverage accuracy (%): 49.68 -> 51.92Improved
Use structure-preserving parameter-scale adaptation instead of truncation (ablation)XPERT · BoolQ/Hellaswag/PIQA/WinoG averageaverage accuracy (%): 49.38 -> 51.92Improved
Select experts using mixed-domain data instead of a single domain (ablation)XPERT · downstream SFTdownstream performanceImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy