Research map / Large language models / Instruction and fine-tuning
Layer Swapping for Zero-Shot Cross-Lingual Transfer in Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Merges a target-language generic-instruction expert and an English math expert by swapping top and bottom transformer layers from the language expert into the math expert, enabling zero-shot cross-lingual math reasoning without in-language math data.
Problem
Fine-tuning LLMs for math reasoning in non-English languages is hard because task-specific instruction data in those languages is scarce, and cross-lingual transfer is often weak.
Approach
Fine-tune separate experts (English math data; target-language generic instruction data) from the same pretrained model, then post hoc replace the top and bottom transformer layers of the math expert with the language expert's layers (with optional averaged transition zones), exploiting the finding that language-specific parameters concentrate in outer layers while math updates concentrate in midd
Key results
- On MGSM, layer swapping with default config reaches Swahili avg 32.4 vs 29.3 for model soup and 29.5 for the math expert; best config Bengali max 45.2 vs 44.4 for best math expert.
- Layer swapping outperforms individual experts and model soup across Swahili, Telugu, Bengali; Japanese average is lower than math expert (40.1 vs 42.7 best config).
- For Swahili, layer swapping best config (32.8 avg) slightly exceeds joint SFT on mixed data (32.1 avg) and its max (37.2) beats joint SFT max (32.8) by 4.4 points.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Swap target-language expert's bottom and top layers into the English math expert | Llama 3.1 8B math and language experts · MGSM 8-shot Swahili | score: 29.5 -> 32.8 | Helped |
| Swap target-language expert's bottom and top layers into the English math expert | Llama 3.1 8B math and language experts · MGSM 8-shot Telugu | score: 20.1 -> 23.0 | Helped |
| Swap target-language expert's bottom and top layers into the English math expert | Llama 3.1 8B math and language experts · MGSM 8-shot Bengali | score: 38.3 -> 38.7 | Helped |
| Swap target-language expert's bottom and top layers into the English math expert | Llama 3.1 8B math and language experts · MGSM 8-shot Japanese | score: 42.7 -> 40.1 | Hurt |
| Use layer swapping instead of model souping | Llama 3.1 8B math and language experts · MGSM 8-shot Swahili | average score: 29.3 -> 32.4 | Helped |
| Use layer swapping instead of model souping | Llama 3.1 8B math and language experts · MGSM 8-shot Telugu | average score: 20.9 -> 22.7 | Helped |
| Use layer swapping instead of model souping | Llama 3.1 8B math and language experts · MGSM 8-shot Bengali | average score: 36.8 -> 37.1 | Helped |
| Use layer swapping instead of model souping | Llama 3.1 8B math and language experts · MGSM 8-shot Japanese | average score: 38.7 -> 38.5 | No clear effect |
| Replace transition-zone averaged layers with math-expert layers (ablation) | Layer swap · MGSM | score | No clear effect |
| Vary the number of swapped layers and transition-zone layers within the tested effective ranges (ablation) | Layer swap · MGSM across Swahili, Telugu, Bengali, and Japanese | score | No clear effect |
| Use more language-expert layers for lower-performing languages such as Telugu (ablation) | Layer swap · MGSM Telugu | score | Helped |
| Remove the first five and last two transformer layers' SFT updates from the math expert (ablation) | Llama 3.1 8B math expert · MGSM | score | No clear effect |
Surprising findings
- Reverting the first five and last two transformer layers of math experts to base values preserved their math performance, and transition zones gave no statistically significant benefit over direct swapping.
- Weight-level merging methods (model soup, TIES-merging) underperform layer-level merging, suggesting granular weight averaging causes interference.
Limitations
- Only tested on Llama 3.1 8B (32 layers) and four target languages; generalization to other models or sizes is unknown.
- Requires minimum target-language capability; unclear how well the model must understand the language.
- Results on Japanese were lower than the math expert, and gains were smaller for higher-resource languages.
Experimental setup
Datasets: Orca-Math Aya Dataset NLLB translation data Inkuba-instruct xP3mt llm-japanese-dataset ichikara-instruction dolly-15k-ja BongChat IndicShareLlama (indic-align) Belebele FLoRes
Benchmarks: MGSM Belebele FLoRes MBPP MMLU
Metrics: accuracy (8-shot MGSM) accuracy (Belebele, MMLU) BLEU (FLoRes) pass@1 (MBPP) average and max over merged pairs
Baselines: Llama 3.1 8B base model soup TIES-merging dataset merging (joint SFT) language expert math expert
Builds on: model souping TIES-merging composable sparse fine-tuning
Closest papers by meaning
- MergeBench: A Benchmark for Merging Domain-Specialized LLMs (NeurIPS 2025 · similarity 0.89)
- Beyond English-Centric Training: How Reinforcement Learning Improves Cross-Lingual Reasoning in LLMs (ICLR 2026 · similarity 0.87)
- Model Merging in Pre-training of Large Language Models (NeurIPS 2025 · similarity 0.87)
- Channel Merging: Preserving Specialization for Merged Experts (AAAI 2025 · similarity 0.87)
- Cross-model Control: Improving Multiple Large Language Models in One-time Training (NeurIPS 2024 · similarity 0.87)
- Universal Cross-Tokenizer Distillation via Approximate Likelihood Matching (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.