AI Paper Atlas

Research map / Architectures and efficiency / Attention and transformers

Quantifying LLM Attention-Head Stability: Implications for Circuit Universality

Karan Bali, Jack Stanley, Praneet Suresh, Danilo Bzdok

ICML 2026 · Architectures and efficiency › Attention and transformers · approach: Transformer expressivity and circuits · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use AdamW instead of Adam for pretraining (ablation)8- and 12-layer MLP GPT models and all attention-only models · attention-head seed stability across refitshead stabilityImproved
use AdamW instead of Adam for pretraining (ablation)2- and 4-layer MLP GPT models · attention-head seed stability across refitshead stabilityNo clear effect
use AdamW instead of Adam for pretraining (ablation)GPT-like models · wikitext-2 validation perplexityvalidation perplexityNo clear effect
use AdamW instead of Adam for pretraining (ablation)GPT-like models · stability gap between most and least stable layerstability gapImproved
use AdamW instead of Adam for pretraining (ablation)GPT-like models · positions of most and least stable layerrelative depth of stability extremesNo clear effect
use AdamW instead of Adam for pretraining (ablation)GPT-like models · residual stream stability (CKA)residual stream CKA stabilityNo clear effect
use AdamW instead of Adam for pretraining (ablation)GPT-like models · attention-head output activation normmean l2 norm of head outputsImproved
increase depth of the model (ablation)GPT-like models · stability gap across layersstability gapHurt
increase prompt length from 5 to 50 tokens (ablation)middle-layer attention heads · layer-wise head stabilityhead stabilityHurt
allow cross-layer matching of heads (ablation)stability metric · head stability across refitshead stabilityImproved
larger mean query-weight norm in a layerlayers of GPT-like models · layer-wise head stabilityhead stabilityHurt
compare residual stream instead of attention headsGPT-like refits · cross-seed stabilitystabilityImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy