AI Paper Atlas

Research map / Large language models / Instruction and fine-tuning

A Single Layer to Explain Them All: Understanding Massive Values in Large Language Models

Zeru Shi, Zhenting Wang, Fan Yang, Qifan Wang, Ruixiang Tang

ICML 2026 · Large language models › Instruction and fine-tuning · approach: Gradient-free adaptation and merging

Read the paper

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add WeMask (mask dimensions with largest RMSNorm weights) at inference, training-freeQwen3-4B + SFT · instruction SFT, 5-benchmark averageaverage accuracy: 64.43 -> 64.91Improved
train with WeMask enabledQwen3-4B · MMLU (instruction SFT)accuracy: 53.77 -> 55.02Improved
apply WeMask with mask rate 1.0 (mask all dimensions) (ablation)Qwen3-4B + SFT, training-free · instruction SFT, 5-benchmark averageaverage accuracy: 64.43 -> 37.89Hurt
mask random dimensions instead of RMSNorm-weight-guided ones (ablation)WeMask · PIQAaccuracy: 81.03 -> 56.02Hurt
mask largest-activation dimensions instead of RMSNorm-weight-guided ones (ablation)WeMask · ARC-Caccuracy: 87.54 -> 29.10Hurt
add WeMask (SFT) during safety SFTQwen3-4B + SFT · Sorry-Benchscore (lower is better): 3.18 -> 1.71Improved
add WeMask (training-free) to SFT modelQwen3-4B + SFT · XSTestaccuracy: 66.22 -> 79.78 (mask rate 1.0); 74.00 at 0.7Improved
add WeMask (SFT, mask rate 1.0) in math SFT (ablation)Qwen3-4B · GSM8Kaccuracy: 20.26 -> 14.59Hurt
add WeMask (training-free) after DPOQwen3-4B + DPO · XSTestaccuracy: 72.30 -> 74.96Improved
add WeMask (training-free) after GRPOQwen3-4B + GRPO · AIME22-24 / Math500accuracy: AIME 7.40 -> 9.27; Math500 43.67 -> 43.47Mixed
add WeMask on Llama-3.1-8B-Instruct (training-free vs SFT baseline)Llama3.1-8B-Instruct + SFT · MMLU 0-shotaccuracy: 48.22 -> 48.15No clear effect
add WeMask (SFT) on Qwen3-8BQwen3-8B + SFT · MMLU 0-shotaccuracy: 60.09 -> 62.96Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy