Research map / Large language models / Language, safety and interpretability

The Same but Different: Structural Similarities and Differences in Multilingual Language Modeling

ICLR 2025 · Large language models › Language, safety and interpretability · approach: Multilingual LLM data and culture · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Uses mechanistic interpretability (path patching and information flow routes) to show that LLMs reuse shared circuits for the same morphosyntactic process across English and Chinese, including in independently trained monolingual models, while language-specific FFNs handle English-only past-tense morphology.

Problem

Whether the internal circuitry of LLMs mirrors linguistic structure: do languages sharing a morphosyntactic process use shared circuits, and do languages needing different processes use different circuits?

Approach

Applies circuit discovery (path patching, information flow routes) to IOI and English-vs-Chinese past-tense tasks, comparing a multilingual model (BLOOM, Qwen) and independently trained English (GPT2-small) and Chinese (CPM-distilled) monolingual models, to test shared versus language-specific circuitry.

Key results

• Pearson correlation of head activation frequency is 0.72 between English and Chinese IOI on BLOOM versus 0.48 between English IOI and English past-tense control.
• GPT2-small and CPM-distilled implement highly similar IOI circuits despite independent training, though CPM lacks Negative Name Mover Heads and has a Copy Suppression Head instead.
• On Qwen2-0.5B, ablating FFN layers 20-24 drops English accuracy from 97.44% to 47.44% while Chinese zero-rank rate barely changes (25.08% to 24.19%).

Measured effects reported in the paper

Surprising findings

• Monolingual English and Chinese models independently trained converge on nearly the same IOI algorithm.
• Chinese past-tense task shows almost no active heads in later layers, yet copy heads are shared across languages and crucial for both.

Limitations

• Results limited to two tasks and small models (BLOOM-560M, Qwen2-0.5B, GPT2-small, CPM distilled).
• Chinese past-tense task lacks minimal pairs, so path patching is done only on English templates.
• Only English and Chinese languages studied; the Pearson correlation analysis is quantitative for only one pair of tasks.

Experimental setup

Datasets: IOI templates (constructed, translated to Chinese) BIG-bench English verb tense dataset

Benchmarks: IOI task past tense task

Metrics: accuracy zero-rank rate activation frequency Pearson correlation rank change after ablation

Baselines: GPT2-small CPM-Generate-distill BLOOM-560M Qwen2-0.5B-instruct

Builds on: path patching indirect object identification circuit information flow routes

Closest papers by meaning

• How do Large Language Models Handle Multilingualism? (NeurIPS 2024 · similarity 0.91)
• The Rise and Down of Babel Tower: Investigating the Evolution Process of Multilingual Code Large Language Model (ICLR 2025 · similarity 0.89)
• Language Model Behavioral Phases are Consistent Across Architecture, Training Data, and Scale (NeurIPS 2025 · similarity 0.87)
• Emergence of a High-Dimensional Abstraction Phase in Language Transformers (ICLR 2025 · similarity 0.87)
• The Emergence of Abstract Thought in Large Language Models Beyond Any Language (NeurIPS 2025 · similarity 0.87)
• Less is More: Local Intrinsic Dimensions of Contextual Language Models (NeurIPS 2025 · similarity 0.87)