Research map / Large language models / Instruction and fine-tuning
A Single Layer to Explain Them All: Understanding Massive Values in Large Language Models
Zeru Shi, Zhenting Wang, Fan Yang, Qifan Wang, Ruixiang Tang
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add WeMask (mask dimensions with largest RMSNorm weights) at inference, training-free | Qwen3-4B + SFT · instruction SFT, 5-benchmark average | average accuracy: 64.43 -> 64.91 | Improved |
| train with WeMask enabled | Qwen3-4B · MMLU (instruction SFT) | accuracy: 53.77 -> 55.02 | Improved |
| apply WeMask with mask rate 1.0 (mask all dimensions) (ablation) | Qwen3-4B + SFT, training-free · instruction SFT, 5-benchmark average | average accuracy: 64.43 -> 37.89 | Hurt |
| mask random dimensions instead of RMSNorm-weight-guided ones (ablation) | WeMask · PIQA | accuracy: 81.03 -> 56.02 | Hurt |
| mask largest-activation dimensions instead of RMSNorm-weight-guided ones (ablation) | WeMask · ARC-C | accuracy: 87.54 -> 29.10 | Hurt |
| add WeMask (SFT) during safety SFT | Qwen3-4B + SFT · Sorry-Bench | score (lower is better): 3.18 -> 1.71 | Improved |
| add WeMask (training-free) to SFT model | Qwen3-4B + SFT · XSTest | accuracy: 66.22 -> 79.78 (mask rate 1.0); 74.00 at 0.7 | Improved |
| add WeMask (SFT, mask rate 1.0) in math SFT (ablation) | Qwen3-4B · GSM8K | accuracy: 20.26 -> 14.59 | Hurt |
| add WeMask (training-free) after DPO | Qwen3-4B + DPO · XSTest | accuracy: 72.30 -> 74.96 | Improved |
| add WeMask (training-free) after GRPO | Qwen3-4B + GRPO · AIME22-24 / Math500 | accuracy: AIME 7.40 -> 9.27; Math500 43.67 -> 43.47 | Mixed |
| add WeMask on Llama-3.1-8B-Instruct (training-free vs SFT baseline) | Llama3.1-8B-Instruct + SFT · MMLU 0-shot | accuracy: 48.22 -> 48.15 | No clear effect |
| add WeMask (SFT) on Qwen3-8B | Qwen3-8B + SFT · MMLU 0-shot | accuracy: 60.09 -> 62.96 | Improved |
Closest papers by meaning
- From Attention to Activation: Unraveling the Enigmas of Large Language Models (ICLR 2025 · similarity 0.90)
- Massive Values in Self-Attention Modules are the Key to Contextual Knowledge Understanding (ICML 2025 · similarity 0.89)
- Exploring Activation Patterns of Parameters in Language Models (AAAI 2025 · similarity 0.89)
- The Tell-Tale Norm: $\ell_2$ Magnitude as a Signal for Reasoning Dynamics in Large Language Models (ICML 2026 · similarity 0.89)
- The Emergence of Abstract Thought in Large Language Models Beyond Any Language (NeurIPS 2025 · similarity 0.89)
- Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calibration (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
