Research map / Architectures and efficiency / Attention and transformers
Revisiting Anisotropy in Language Transformers: The Geometry of Learning Dynamics
Raphael Bernas, Fanny Jourdan, Antonin Poché, CELINE HUDELOT
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use causal decoder rather than encoder model for frequency-induced variance collapse | Pythia and SmolLM2 token embeddings · Embedding trajectories across checkpoints | Pearson correlation of log-frequency vs distance to centroid: rho ≈ -0.5 | Improved |
| EuroBERT-610m shows no frequency concentration in mean statistic | EuroBERT-610m · Embedding trajectories across checkpoints | Pearson correlation: rho ≈ 0.1 (610m), ≈ -0.2 (210m) | No clear effect |
| project onto activation-derived tangent subspace instead of matched-rank normal subspace (ablation) | gradient energy · Early training, first layer input, EuroBERT-210m | gradient energy ratio: E_r 7.42 | Improved |
| project onto tangent subspace instead of normal subspace (ablation) | Pythia-160m gradients · Early training, first layer input | gradient energy ratio: E_r 545 | Improved |
| remove tangent proxy rather than matched normal basis (ablation) | gradient covariance isotropy · Early training, pythia-1b first layer input | IsoScore* change (%): 85.4 vs -0.198 | Improved |
| evaluate in late rather than early training | tangent alignment of gradient energy · EuroBERT-210m first layer input | energy ratio E_r: 7.42 -> 4.05 | Hurt |
| evaluate in late rather than early training | tangent alignment of gradient energy · pythia-1b first layer input | energy ratio E_r: 323 -> 96.7 | Hurt |
| tangent removal in late phase on Pythia embedding probes | IsoScore* recovery · pythia-1b E0 in, late phase | IsoScore* change (%): -0.01 | Hurt |
| use encoder models (EuroBERT) instead of decoders | tangent alignment · Cross-model Table 1 | alignment strength | Hurt |
| move from early/middle layers to final layers | tangent bias · Cross-model Table 1 | alignment strength | Hurt |
Closest papers by meaning
- Emergence of a High-Dimensional Abstraction Phase in Language Transformers (ICLR 2025 · similarity 0.90)
- Tracing the Representation Geometry of Language Models from Pretraining to Post-training (NeurIPS 2025 · similarity 0.89)
- Geometry of Decision Making in Language Models (NeurIPS 2025 · similarity 0.87)
- Transformers Represent Belief State Geometry in their Residual Stream (NeurIPS 2024 · similarity 0.86)
- Less is More: Local Intrinsic Dimensions of Contextual Language Models (NeurIPS 2025 · similarity 0.86)
- Invariant Features in Language Models: Geometric Characterization and Model Attribution (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
