Research map / Large language models / Instruction and fine-tuning
XCOT: Cross-lingual Instruction Tuning for Cross-lingual Chain-of-Thought Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes xCoT, a cross-lingual instruction tuning framework that combines code-switched in-context examples, random online translate-then-answer CoT, and token-level cross-lingual distillation to improve multilingual chain-of-thought math reasoning.
Problem
Chain-of-thought reasoning is strong in English but weak in low-resource languages; prompt-based methods do not exploit cross-lingual representation alignment from supervised fine-tuning.
Approach
Instruction-tunes LLMs on multilingual GSM8K-style queries with English responses, mixes source and target language fragments in in-context examples (xICL), randomly inserts an intermediate translation step during training (Random-CoT), and distills the high-resource answer distribution into low-resource queries (xDistill).
Key results
- On MGSM with Llama-2-13B, xCoT reaches 51.5 average accuracy across 11 languages, versus 47.6 for MathOctopus.
- On MGSM with Llama-2-7B, xCoT reaches 47.7 average accuracy, versus 41.9 for MathOctopus under the same setting.
- On MSVAMP, the paper reports 42.9 average at 7B level, which the text says is the best across languages; ablation shows the full model improves the Llama-2-7B base from 30.1 to 47.7 on MGSM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use xCoT instead of MathOctopus | Llama-2-7B · MGSM | accuracy (Avg.): 41.9 -> 47.7 | Helped |
| Use xCoT instead of MathOctopus | Llama-2-13B · MGSM | accuracy (Avg.): 47.6 -> 51.5 | Helped |
| Use xCoT instead of MathOctopus | Llama-2-7B · MSVAMP | accuracy: 42.4 -> 42.9 | Helped |
| Add xICL code-switched context (ablation) | Llama-2-7B trained on xCoT-Instruct · MGSM | accuracy (Avg.): 30.1 -> 34.8; +4.7 points | Helped |
| Add multilingual sampling (mSampling) (ablation) | Llama-2-7B + xICL · MGSM | accuracy (Avg.): 34.8 -> 40.6; +5.8 points | Helped |
| Add Random-CoT (ablation) | Llama-2-7B + xICL + mSampling · MGSM | accuracy (Avg.): 40.6 -> 46.7 | Helped |
| Add cross-lingual distillation (xDistill) (ablation) | Llama-2-7B + xICL + mSampling + Random-CoT · MGSM | accuracy (Avg.): 46.7 -> 47.7 | Helped |
| Use low-resource-to-English and high-resource-to-high-resource translation directions instead of all-language-to-English (ablation) | Random-CoT on Llama-2-7B · MGSM | accuracy (Avg.): 40.6 -> 44.3 | Helped |
| Use all-language-to-all-language translation direction instead of all-language-to-English (ablation) | Random-CoT on Llama-2-7B · MGSM | accuracy (Avg.): 40.6 -> 45.7 | Helped |
| Use all-language-to-high-resource translation direction instead of all-language-to-English (ablation) | Random-CoT on Llama-2-7B · MGSM | accuracy (Avg.): 40.6 -> 47.7 | Helped |
| Use code-switched context instead of English context (ablation) | Bloom-7B · MGSM | accuracy (Avg.): 29.9 -> 30.1 | Helped |
| Use code-switched context instead of native context (ablation) | Bloom-7B · MGSM | accuracy (Avg.): 29.2 -> 30.1 | Helped |
Surprising findings
- Bloom-7b1 with xCoT reaches only 30.1 average on MGSM, far below Llama-2-7B based xCoT (47.7).
Limitations
- Evaluation limited to math word problem benchmarks (MGSM, MSVAMP).
- Relies on a multilingual translator to build training data, with unspecified translation quality.
- Bloom-based results are much weaker than Llama-2-based results, suggesting dependence on the base model.
Experimental setup
Datasets: GSM8K MGSM MSVAMP xCoT-Instruct
Benchmarks: MGSM MSVAMP
Metrics: accuracy
Baselines: GPT-3.5 Native-CoT GPT-4 Native-CoT Llama-2 RFT MAmmoTH WizardMath MathOctopus
Builds on: chain-of-thought prompting RFT MathOctopus MAmmoTH
Closest papers by meaning
- AdaMCoT: Rethinking Cross-Lingual Factual Reasoning Through Adaptive Multilingual Chain-of-Thought (AAAI 2026 · similarity 0.92)
- Reasoning Transfer for an Extremely Low-Resource and Endangered Language: Bridging Languages Through Sample-Efficient Language Understanding (AAAI 2026 · similarity 0.92)
- Beyond English-Centric Training: How Reinforcement Learning Improves Cross-Lingual Reasoning in LLMs (ICLR 2026 · similarity 0.89)
- Pushing on Multilingual Reasoning Models with Language-Mixed Chain-of-Thought (ICLR 2026 · similarity 0.89)
- Incentivizing Reasoning for Advanced Instruction-Following of Large Language Models (NeurIPS 2025 · similarity 0.89)
- Long Chain-of-Thought Reasoning Across Languages (ICLR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.