Research map / Large language models / Language, safety and interpretability
Focusing on Language: Revealing and Exploiting Language Attention Heads in Multilingual Large Language Models
Xin Liu, Qiyang Song, Qihang Zhou, Haichao Du, Shaowen Xu, Wenbo Jiang, Weijuan Zhang, Xiaoqi Jia
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train language-specific head mask (14-20 params) on top 2% LAHIS heads | Mistral-7B · XQuAD (avg) | accuracy: 22.53 -> 29.03 | Improved |
| train language-specific head mask | Aya-23-8B · XQuAD (avg) | accuracy: 55.28 -> 61.10 | Improved |
| train language-specific head mask | Llama-3.2-3B · XQuAD (avg) | accuracy: 32.98 -> 36.78 | Improved |
| use LAHIS-selected heads instead of random heads for the mask (ablation) | Aya-23-8B · XQuAD (avg) | accuracy: 56.15 -> 61.10 | Improved |
| use random head mask instead of vanilla (ablation) | Aya-23-8B · XQuAD (avg) | accuracy: 55.28 -> 56.15 | No clear effect |
| deactivate language-general heads (ablation) | Aya-23-8B · XL-Sum (avg) | BERTScore F1: 78.5 -> 66.2 | Hurt |
| deactivate language-general heads (ablation) | Llama-3.2-3B · XL-Sum (avg) | BERTScore F1: 66.3 -> 49.6 | Hurt |
| deactivate the same number of random heads (ablation) | Llama-3.2-3B · XL-Sum (avg) | BERTScore F1: 66.3 -> 66.8 | No clear effect |
| deactivate language-specific heads (ablation) | Aya-23-8B · Wikipedia PPL (zh) | perplexity: 8.66 -> 14.1 | Hurt |
| deactivate language-specific heads (ablation) | Mistral-7B · Wikipedia PPL (th) | perplexity: 4.35 -> 7.70 | Hurt |
| deactivate random heads of same count (ablation) | Aya-23-8B · Wikipedia PPL (zh) | perplexity: 8.66 -> 8.94 | No clear effect |
| deactivate English heads to cut off-target English output | Mistral-7B · XL-Sum Vietnamese | language accuracy: 0.35 -> 1.00 | Improved |
Closest papers by meaning
- How do Large Language Models Handle Multilingualism? (NeurIPS 2024 · similarity 0.90)
- Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calibration (ICML 2024 · similarity 0.90)
- Unveiling Visual Perception in Language Models: An Attention Head Analysis Approach (CVPR 2025 · similarity 0.90)
- Order-Level Attention Similarity Across Language Models: A Latent Commonality (NeurIPS 2025 · similarity 0.88)
- The Same but Different: Structural Similarities and Differences in Multilingual Language Modeling (ICLR 2025 · similarity 0.88)
- Supervised Knowledge Makes Large Language Models Better In-context Learners (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
