Research map / Large language models / Language, safety and interpretability
How do Large Language Models Handle Multilingualism?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes the MWork hypothesis that LLMs process multilingual queries via English-centric latent reasoning, and uses PLND, a parallel gradient-free neuron detection method, to locate language-specific neurons for deactivation and fine-tuning experiments.
Problem
How LLMs internally handle multilingual inputs; existing interpretability work focuses on English and ignores multilingual behavior.
Approach
Decoding per-layer hidden states shows an English-centric middle stage; a three-stage workflow (understand, task-solve with attention for reasoning and FFN for multilingual knowledge, generate) is hypothesized and verified by deactivating language-specific neurons found with PLND, which needs no labeled data or parameter updates. Fine-tuning only these neurons on a few hundred documents improves a
Key results
- Deactivating language-specific neurons (about 0.13% of neurons) drops XLSum ROUGE-L to near zero across languages, while random deactivation leaves performance unchanged (Vicuna avg 26.6 to 0.37).
- Fine-tuning language-specific neurons with a few hundred Wikipedia documents improves high-resource languages by 3.6% on average and low-resource languages by 2.3% (per abstract).
- Fine-tuning on a non-English corpus also improves English by about 3.7% on average across tasks in the high-resource experiments.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| deactivate language-specific neurons (ablation) | Vicuna · XLSum average across French, Chinese, Spanish, and Russian | ROUGE-L: 26.6 -> 0.37 | Hurt |
| deactivate an equivalent number of randomly selected neurons (ablation) | Vicuna · XLSum average across four languages | ROUGE-L: 26.6 -> 26.7 | No clear effect |
| deactivate language-specific neurons (ablation) | Mistral · XLSum average across French, Chinese, Spanish, and Russian | ROUGE-L: 25.8 -> 0.21 | Hurt |
| deactivate an equivalent number of randomly selected neurons (ablation) | Mistral · XLSum average across four languages | ROUGE-L: 25.8 -> 25.7 | No clear effect |
| fine-tune language-specific neurons using 400 documents | Mistral-7B-v0.1 · MGSM, high-resource languages | accuracy: +3.4% | Helped |
| fine-tune language-specific neurons using 400 documents | Mistral-7B-v0.1 · XQuAD, high-resource languages | accuracy: +4.4% | Helped |
| fine-tune language-specific neurons using 400 documents | Mistral-7B-v0.1 · X-CSQA, high-resource languages | accuracy: +4.3% | Helped |
| fine-tune language-specific neurons using 400 documents | Mistral-7B-v0.1 · XLSum, high-resource languages | ROUGE-L: +2.3% | Helped |
| fine-tune language-specific neurons with 800 rather than 400 documents (ablation) | Mistral-7B-v0.1 · high-resource multilingual tasks | performance | Hurt |
| fine-tune language-specific neurons for a target language | Mistral-7B-v0.1 · low-resource languages across tasks | average performance: +2.2% | Helped |
| fine-tune language-specific neurons without English training data | Mistral-7B-v0.1 · English across all tasks | performance: +3.5% | Helped |
Surprising findings
- Chinese Llama shows no language-specific neurons, with Chinese dominating reasoning and knowledge extraction across all languages.
- Fine-tuning with 800 documents performs worse than with 400, attributed to overfitting to Wikipedia.
Limitations
- Experiments limited to a few 7B models and a small set of high- and low-resource languages.
- Language classification of decoded tokens uses cld3 and discards unreliable detections, which may bias the layer-wise ratios.
Experimental setup
Datasets: OSCAR Wikipedia XLSum XQuAD MGSM X-CSQA
Benchmarks: XLSum XQuAD MGSM X-CSQA
Metrics: accuracy ROUGE-L
Baselines: random neuron deactivation
Builds on: Transformer feed-forward key-value memory logit lens
Closest papers by meaning
- The Emergence of Abstract Thought in Large Language Models Beyond Any Language (NeurIPS 2025 · similarity 0.91)
- The Same but Different: Structural Similarities and Differences in Multilingual Language Modeling (ICLR 2025 · similarity 0.91)
- Enhancing Non-English Capabilities of English-Centric Large Language Models Through Deep Supervision Fine-Tuning (AAAI 2025 · similarity 0.90)
- Identifying Query-Relevant Neurons in Large Language Models for Long-Form Texts (AAAI 2025 · similarity 0.89)
- When Less Language is More: Language-Reasoning Disentanglement Makes LLMs Better Multilingual Reasoners (NeurIPS 2025 · similarity 0.89)
- The Rise and Down of Babel Tower: Investigating the Evolution Process of Multilingual Code Large Language Model (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.