AI Paper Atlas

Research map / Architectures and efficiency / Attention and transformers

Revisiting Anisotropy in Language Transformers: The Geometry of Learning Dynamics

Raphael Bernas, Fanny Jourdan, Antonin Poché, CELINE HUDELOT

ICML 2026 · Architectures and efficiency › Attention and transformers · approach: Transformer expressivity and learnability · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use causal decoder rather than encoder model for frequency-induced variance collapsePythia and SmolLM2 token embeddings · Embedding trajectories across checkpointsPearson correlation of log-frequency vs distance to centroid: rho ≈ -0.5Improved
EuroBERT-610m shows no frequency concentration in mean statisticEuroBERT-610m · Embedding trajectories across checkpointsPearson correlation: rho ≈ 0.1 (610m), ≈ -0.2 (210m)No clear effect
project onto activation-derived tangent subspace instead of matched-rank normal subspace (ablation)gradient energy · Early training, first layer input, EuroBERT-210mgradient energy ratio: E_r 7.42Improved
project onto tangent subspace instead of normal subspace (ablation)Pythia-160m gradients · Early training, first layer inputgradient energy ratio: E_r 545Improved
remove tangent proxy rather than matched normal basis (ablation)gradient covariance isotropy · Early training, pythia-1b first layer inputIsoScore* change (%): 85.4 vs -0.198Improved
evaluate in late rather than early trainingtangent alignment of gradient energy · EuroBERT-210m first layer inputenergy ratio E_r: 7.42 -> 4.05Hurt
evaluate in late rather than early trainingtangent alignment of gradient energy · pythia-1b first layer inputenergy ratio E_r: 323 -> 96.7Hurt
tangent removal in late phase on Pythia embedding probesIsoScore* recovery · pythia-1b E0 in, late phaseIsoScore* change (%): -0.01Hurt
use encoder models (EuroBERT) instead of decoderstangent alignment · Cross-model Table 1alignment strengthHurt
move from early/middle layers to final layerstangent bias · Cross-model Table 1alignment strengthHurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy