Research map / Continual learning / Continual fine-tuning of LLMs

MoE-LPR: Multilingual Extension of Large Language Models Through Mixture-of-Experts with Language Priors Routing

AAAI 2025 · Continual learning › Continual fine-tuning of LLMs · approach: Continual pretraining of language models · 9 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MoE-LPR, a two-stage post-pretraining method that upcycles Qwen1.5-1.8B into a frozen-original MoE with new experts trained on expanded languages, then trains only the router with a language-priors routing loss on under 1% replay data to reduce forgetting of original languages.

Problem

Post-pretraining LLMs on new low-resource languages improves those languages but causes catastrophic forgetting of high-resource original languages; existing methods trade off expansion against forgetting.

Approach

Upcycle a dense LLM into an MoE by copying the frozen original FFN as expert 0 plus new experts trained on expanded-language data; then in a review stage train only the router with a cross-entropy loss that routes original-language tokens to the frozen expert 0 (language priors routing), using replay data under 1% of the corpus.

Key results

• MoE-LPR reaches 45.07 average on expanded languages (Greek, Hungarian, Turkish) vs 42.37 for LoRAMoE and 34.36 for the base model.
• On original languages (English, Chinese, Spanish), MoE-LPR averages 52.12 vs 53.97 for the base model and 51.24 for LoRAMoE, recovering about 96.6% of original performance.
• Review-with-LPR stage raises original-language average by about 5 points over MoE-LPR without review (47.14 to 52.12 in the ablation table) without hurting expanded languages.

Measured effects reported in the paper

Surprising findings

• Full fine-tuning with under 1% replay still loses about 4 points on original languages, while more than 25% replay was reported to be needed for stronger recovery in prior work cited by the authors.
• Expert-copy initialization matters more for expanded-language performance than for original-language retention; random initialization of new experts causes a large drop in expanded languages with little original-language change.

Limitations

• Evaluated only on a single base model (Qwen1.5-1.8B) and three expanded languages.
• Original-language performance still drops relative to the base model (52.12 vs 53.97), so forgetting is reduced rather than eliminated.
• Requires training from a dense checkpoint with frozen original FFN, so total parameter count grows with the number of experts.

Experimental setup

Datasets: CulturalX SlimPajama SkyPile-150B

Benchmarks: ARC-Challenge MMLU HellaSwag Belebele FLORES-101 CMMLU

Metrics: accuracy COMET

Baselines: Full Fine-tuning LoRA MoE LoRAMoE LLaMA-Pro

Builds on: Mixtral sparse upcycling LoRAMoE LLaMA-Pro Lifelong-MoE

Closest papers by meaning

• Model Merging in Pre-training of Large Language Models (NeurIPS 2025 · similarity 0.88)
• Pruning and Distilling Mixture-of-Experts into Dense Language Models (NeurIPS 2026 · similarity 0.88)
• Mixture-of-Experts Meets Instruction Tuning: A Winning Combination for Large Language Models (ICLR 2024 · similarity 0.88)
• Advancing Expert Specialization for Better MoE (NeurIPS 2025 · similarity 0.88)
• FlexMoE: One-for-All Nested Intra-Expert Pruning for MoE Language Models (NeurIPS 2026 · similarity 0.88)
• MoEQuant: Enhancing Quantization for Mixture-of-Experts Large Language Models via Expert-Balanced Sampling and Affinity Guidance (ICML 2025 · similarity 0.87)