Research map / Large language models / Language, safety and interpretability
MindMerger: Efficiently Boosting LLM Reasoning in non-English Languages
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MindMerger, which merges a frozen multilingual encoder (mT5-xl) with a frozen LLM via a trained mapping layer in a two-stage scheme, improving multilingual reasoning especially for low-resource languages.
Problem
LLM reasoning in non-English languages, especially low-resource ones, lags behind English; existing relearning and translate-then-replace methods underuse the LLM's built-in reasoning and language understanding.
Approach
Feed the LLM the undecoded hidden states of a multilingual encoder through a trainable mapping layer, concatenated with the LLM's own query embeddings (augmentation rather than replacement), trained first on general bilingual translation pairs (mapping stage) and then on query-translation task data (augmentation stage), with both base models frozen.
Key results
- MindMerger-Soft achieves 57.3% average accuracy on MGSM, at least 6.7 points above the best baseline, with 53.5% on low-resource languages (8.0+ point gain) without updating LLM parameters.
- On MSVAMP, MindMerger-Soft reaches 60.4% average, at least 3.2 points above the best baseline.
- On X-CSQA MindMerger-Soft averages 61.0% and on XNLI 78.4%, with MindMerger-Soft outperforming all baselines on both.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MindMerger-Soft instead of the strongest reported baseline | Llama 2-7B · MGSM | average accuracy: 50.6 -> 57.3 | Helped |
| Use MindMerger-Soft instead of the strongest reported baseline | Llama 2-7B · MSVAMP | average accuracy: 57.2 -> 60.4 | Helped |
| Use MindMerger-Soft instead of the strongest reported baseline | Llama 2-7B · X-CSQA | average accuracy: 56.9 -> 61.0 | Helped |
| Use MindMerger-Soft instead of the strongest reported baseline | Llama 2-7B · XNLI | average accuracy: 76.5 -> 78.4 | Helped |
| Add the mapping stage (ablation) | MindMerger-Soft · MGSM | low-resource average accuracy: 31.5 -> 51.7; 39.1 -> 53.5 | Helped |
| Add the mapping stage (ablation) | MindMerger-Soft · MSVAMP | average accuracy: 50.8 -> 59.1; 53.1 -> 60.4 | Helped |
| Add the augmentation stage (ablation) | MindMerger-Soft · MGSM | average accuracy: 42.5 -> 56.3; 42.6 -> 57.3 | Helped |
| Add the augmentation stage (ablation) | MindMerger-Soft · MSVAMP | average accuracy: 49.1 -> 59.1; 52.4 -> 60.4 | Helped |
| Add the original LLM query representation alongside multilingual features (ablation) | MindMerger-Soft · MGSM | average accuracy: 54.6 -> 56.3; 55.9 -> 57.3 | Helped |
| Add the original LLM query representation alongside multilingual features (ablation) | MindMerger-Soft · MSVAMP | average accuracy: 55.6 -> 59.1; 57.3 -> 60.4 | Helped |
| Replace 2-layer MLP mapping with linear mapping (ablation) | MindMerger-Soft · MGSM | average accuracy: 57.3 -> 55.3 | Hurt |
| Replace 2-layer MLP mapping with QFormer (ablation) | MindMerger-Soft · MGSM | average accuracy: 57.3 -> 22.8 | Hurt |
Surprising findings
- Replacing the LLM's own query embeddings with the multilingual encoder output (no augmentation) drops performance, and the mapping-stage ablation shows low-resource gains depend heavily on general bilingual pairs.
- A QFormer mapping layer, popular in vision model merging, performs very poorly (22.8% average on MGSM).
Limitations
- Evaluation focuses on math, commonsense and NLI tasks; extension to other skills such as code generation is left for future work.
- Relies on the LLM's own multilingual capacity; the paper notes that better results may require enhancing built-in capabilities when training data scales beyond 30K per language.
- Experiments use a limited set of LLMs (Llama 2-7B, MetaMath variants) and a single LLM family for most results.
Experimental setup
Datasets: Lego-MT MetaMathQA MultiNLI X-CSQA OpenBookQA ARC QASC XNLI dev set MGSM translated queries M2M100-1.2B translations
Benchmarks: MGSM MSVAMP X-CSQA XNLI Flores-101
Metrics: accuracy average accuracy across low-resource and high-resource langu spBLEU cosine similarity Recall@1
Baselines: MonoReason MultiReason-SFT MultiReason-Lora QAlign LangBridge Translate-En
Builds on: LangBridge QAlign MetaMath
Closest papers by meaning
- When Less Language is More: Language-Reasoning Disentanglement Makes LLMs Better Multilingual Reasoners (NeurIPS 2025 · similarity 0.91)
- Exploring the Limits of Sub-Billion Language Model Reasoners with Open Training Recipes (ICLR 2026 · similarity 0.90)
- Knowledge Fusion of Large Language Models (ICLR 2024 · similarity 0.90)
- Reasoning Transfer for an Extremely Low-Resource and Endangered Language: Bridging Languages Through Sample-Efficient Language Understanding (AAAI 2026 · similarity 0.90)
- Pushing on Multilingual Reasoning Models with Language-Mixed Chain-of-Thought (ICLR 2026 · similarity 0.89)
- INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledge (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.