Research map / Large language models / Language, safety and interpretability

When Less Language is More: Language-Reasoning Disentanglement Makes LLMs Better Multilingual Reasoners

NeurIPS 2025 · Large language models › Language, safety and interpretability · approach: Multilingual LLM data and culture · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Training-free inference-time projection that removes language-specific subspace components from hidden states, improving multilingual reasoning across 10 open-weight LLMs and 11 languages.

Problem

Multilingual reasoning in LLMs lags strongly in low- and mid-resource languages; the paper asks whether language and reasoning representations are separable and whether suppressing language-specific signals can close the gap.

Approach

Identify a language-specific subspace from mean final-token embeddings of multilingual inputs via an orthogonal language-agnostic/language-specific decomposition (SVD), then project it out of hidden states at middle layers during inference, with no weight updates.

Key results

• Language-reasoning disentanglement improves MGSM average accuracy for all 10 models, e.g., Qwen-2.5-Instruct-7B from 69.82 to 70.76 and R1-Distill-LLaMA-8B from 44.69 to 48.19.
• On XWinograd, Qwen-2.5-Instruct-3B rises from 65.07 to 70.18 average accuracy, and R1-Distill-Qwen-7B on M-MMLU from 41.63 to 45.50.
• In a comparison on Qwen-2.5-3B-Instruct, the training-free method (59.02 MGSM avg) outperforms SFT (49.09) and is comparable to PPO RL (60.33).

Measured effects reported in the paper

Surprising findings

• Supervised fine-tuning on machine-translated MATH data degraded multilingual MGSM performance (49.09 vs. 55.78 baseline), which the authors attribute to translation noise and distillation difficulty for a small model.
• After projection, non-English representations converge toward English rather than forming their own anchor, even in bilingual Qwen models.

Limitations

• Upper-layer interventions substantially reduce language fidelity, forcing outputs toward English.
• Hyperparameter (lambda and layer range) chosen by grid search on evaluation benchmarks; low- and mid-resource languages show trade-offs between reasoning gains and fidelity.
• Post-training comparison limited to Qwen2.5-3B-Instruct and a single math dataset due to compute constraints.

Experimental setup

Datasets: MGSM XWinograd M-MMLU MATH (Google Translate multilingual version) PolyMath

Benchmarks: MGSM XWinograd M-MMLU PolyMath

Metrics: accuracy language fidelity

Baselines: Qwen-2.5-Instruct SFT Qwen-2.5-Instruct PPO RL

Builds on: Linear Representation Hypothesis vLLM PPO LLaMA-Factory

Closest papers by meaning

• AdaMCoT: Rethinking Cross-Lingual Factual Reasoning Through Adaptive Multilingual Chain-of-Thought (AAAI 2026 · similarity 0.92)
• Beyond English-Centric Training: How Reinforcement Learning Improves Cross-Lingual Reasoning in LLMs (ICLR 2026 · similarity 0.91)
• MindMerger: Efficiently Boosting LLM Reasoning in non-English Languages (NeurIPS 2024 · similarity 0.91)
• Reasoning Transfer for an Extremely Low-Resource and Endangered Language: Bridging Languages Through Sample-Efficient Language Understanding (AAAI 2026 · similarity 0.90)
• Pushing on Multilingual Reasoning Models with Language-Mixed Chain-of-Thought (ICLR 2026 · similarity 0.90)
• Calibrating Reasoning in Language Models with Internal Consistency (NeurIPS 2024 · similarity 0.89)