Research map / Large language models / Retrieval and knowledge

Knowledge Localization in Mixture-of-Experts LLMs Using Cross-Lingual Inconsistency

NeurIPS 2026 · Large language models › Retrieval and knowledge · approach: Knowledge editing in LLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Uses cross-lingual inconsistency in factual QA as a contrastive signal over MoE router activations to localize a small set of experts for knowledge, validated by causal ablation on GLM-4.5-Air and Qwen3-30B-A3B.

Problem

Localizing which components of mixture-of-experts LLMs are responsible for retrieving a specific factual piece of knowledge, which is hard because MoE routing is sparse, discrete, and distributed.

Approach

Treat languages where the model answers a factual question correctly versus incorrectly as natural contrastive conditions; apply a one-tailed Mann-Whitney U test with a median-difference threshold to the averaged router weights (after excluding top/bottom layers, blacklisting generalist experts, and subtracting language means) to identify question-specific experts, then validate by deactivating th

Key results

• Deactivating about 20 experts (avg. 20.4 per question) on GLM-4.5-Air yields a 48.0% ablation success rate and 44.1% rate difference, versus about 7% for random question-shuffling.
• On Qwen3-30B-A3B-Instruct-2507, XICI achieves a 35.5% ablation success rate and 32.0% rate difference, versus 7.4% and 3.6% for question-shuffling baseline.
• Using English paraphrases instead of cross-lingual translations lowers ablation success (GLM 34.5% vs 50.1%; Qwen 18.2% vs 34.5%).

Surprising findings

• Activating (boosting router logits of) identified experts did not improve correctness, suggesting identified experts are necessary but not sufficient.
• Spurious gain rates of 3-5% appear even in random baselines, higher than run-to-run variance, indicating redundancy in MoE knowledge storage.

Limitations

• Requires cross-lingual inconsistency; questions the model always answers correctly or always incorrectly cannot be analyzed.
• Ablation success rate is far from 100% and random baselines show non-negligible disruption; no direct comparison to prior expert-localization baseline.
• Does not establish that identified experts store the knowledge or what sub-function they perform.

Experimental setup

Datasets: ECLeKTic MultiLoKo Global MMLU prehistory

Benchmarks: ECLeKTic MultiLoKo Global MMLU prehistory

Metrics: ablation success rate spurious gain rate rate difference

Baselines: Random Expert Set of Same Size Random Question-Shuffling English paraphrase variant of XICI

Builds on: expert deactivation (router logit minimization) Mann-Whitney U test

Closest papers by meaning

• KScope: A Framework for Characterizing the Knowledge Status of Language Models (NeurIPS 2025 · similarity 0.89)
• Knowledge Localization: Mission Not Accomplished? Enter Query Localization! (ICLR 2025 · similarity 0.88)
• Physics of Language Models: Part 3.1, Knowledge Storage and Extraction (ICML 2024 · similarity 0.88)
• INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledge (ICLR 2025 · similarity 0.87)
• Mixture of In-Context Experts Enhance LLMs' Long Context Awareness (NeurIPS 2024 · similarity 0.87)
• Your Mixture-of-Experts LLM Is Secretly an Embedding Model for Free (ICLR 2025 · similarity 0.87)