AI Paper Atlas

Research map / Label-efficient and robust learning / Recognition and long-tail

Singular Vectors of Attention Heads Align with Features

Gabriel Franco, Carson Loughridge, Mark Crovella

ICML 2026 · Label-efficient and robust learning › Recognition and long-tail · approach: Contrastive representation theory · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add an attention head trained on a feature pair to the toy autoencodertoy model (20 features, D=10) · toy modelsingular vector-feature alignmentImproved
increase feature anisotropy up to the range seen in GPT-2 (ablation)toy model with frozen features · anisotropy sweepmean cosine similarity: mean cosine similarity above 0.75No clear effect
anti-whiten singular vectors using the feature covariance (ablation)SVF alignment under anisotropy · anisotropy sweepalignment (avg cos sim)Improved
randomly rotate U and V singular vector matrices (keeping spectrum) (ablation)Pythia-160M IOI heads · sparse attention decomposition, S(v)decomposition sparsenessHurt
train the model longer (checkpoints)Pythia-160M IOI circuit heads · IOI headsnumber of singular vectors to reconstruct relative attention: typically only 1-4 singular vectors neededImproved
present feature pairs of interest in tokens vs absent (ablation)toy model attention decomposition · toy modeldecomposition sparsenessImproved
add rotary positional encoding to the toy model (ablation)toy model · toy model with RoPESVF alignmentNo clear effect
have more features of interest than head dimension (ablation)toy model · over-capacity regimeSVF alignmentMixed

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy