Research map / Large language models / Retrieval and knowledge
The Rise and Down of Babel Tower: Investigating the Evolution Process of Multilingual Code Large Language Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes the Babel Tower Hypothesis describing how code LLMs move from a dominant-language knowledge system to language-specific systems during pre-training, and uses this to build a re-weighted multilingual code pre-training corpus that outperforms the original.
Problem
How multilingual capabilities of LLMs evolve during pre-training is not well understood; prior mechanistic work only studies fully pre-trained models. The paper also asks how to set per-language data proportions for optimal multilingual code performance.
Approach
Tracks a Python-monolingual code model as it learns new programming languages (PHP, C#, Go, C++) and finds a translation, transition, and stabilization sequence; probes internal states with logit-lens working-language identification and language-transferring neurons (LAPE); then estimates optimal per-language token counts from training loss and uses them to rebalance the pre-training corpus.
Key results
- Optimized-1.3B corpus raises average HumanEval+MBXP-averaged AVE from 49.81 (Original-1.3B) to 51.97 and HumanEval AVE from 25.03 to 26.61.
- Optimized-6.7B improves HumanEval AVE from 38.31 (Original-6.7B) to 41.96 and MBXP AVE from 64.34 to 66.14.
- Increasing PHP tokens initially improves HumanEval performance and then declines, so a larger new-language corpus does not necessarily help.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the optimized rather than original pre-training corpus | 1.3B multilingual code LLM · HumanEval AVE | Pass@1: 25.03 -> 26.61 | Helped |
| Use the optimized rather than original pre-training corpus | 1.3B multilingual code LLM · MBXP AVE | Pass@1: 49.81 -> 51.97 | Helped |
| Use the optimized rather than original pre-training corpus | 6.7B multilingual code LLM · HumanEval AVE | Pass@1: 38.31 -> 41.96 | Helped |
| Use the optimized rather than original pre-training corpus | 6.7B multilingual code LLM · MBXP AVE | Pass@1: 64.34 -> 66.14 | Helped |
| Increase the amount of PHP data in the mixed pre-training corpus (ablation) | Python monolingual LLM continually pre-trained with PHP · HumanEval PHP | performance | Mixed |
| Increase the amount of PHP data in the mixed pre-training corpus (ablation) | Python monolingual LLM continually pre-trained with PHP · Final internal state | PHP working-language proportion | Helped |
| Increase the amount of PHP data in the mixed pre-training corpus (ablation) | Python monolingual LLM continually pre-trained with PHP · Final internal state | PHP language-transferring neurons | Hurt |
Surprising findings
- For many languages, reusing the dominant Python knowledge system performs better than building a new language system from more of its own data.
- Adding more PHP data beyond an optimum lowers PHP performance, indicating that using all collected tokens can hurt.
Limitations
- Evaluation limited to code LLMs and programming languages; generalization to natural languages is untested.
- Corpus-size estimation relies on training loss as a proxy for system proportion and a token-share heuristic for data distribution.
- C++ does not follow the Python-dependence pattern, so the method is not uniformly applicable.
Experimental setup
Datasets: GitHub code corpus (PHP, Go, C#, C++, Python) HumanEval MBXP
Benchmarks: HumanEval MBXP
Metrics: Pass@1 proportion of working language number of language transferring neurons average performance (AVE)
Baselines: StarCoder-1B StarCoder-3B DeepseekCoder-1.3B StarCoder-7B StarCoder2-7B CodeLlama-7B DeepseekCoder-6.7B Original corpus pre-training
Builds on: logit lens LAPE language-specific neurons HumanEval MBXP
Closest papers by meaning
- The Same but Different: Structural Similarities and Differences in Multilingual Language Modeling (ICLR 2025 · similarity 0.89)
- How do Large Language Models Handle Multilingualism? (NeurIPS 2024 · similarity 0.89)
- Law of the Weakest Link: Cross Capabilities of Large Language Models (ICLR 2025 · similarity 0.88)
- Exploring Polyglot Harmony: On Multilingual Data Allocation for Large Language Models Pretraining (NeurIPS 2025 · similarity 0.88)
- KoLA: Carefully Benchmarking World Knowledge of Large Language Models (ICLR 2024 · similarity 0.88)
- LLM Augmented LLMs: Expanding Capabilities through Composition (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.