Research map / Architectures and efficiency / Attention and transformers
Modelling Attention with Aitchison Geometry: Token Distinguishability and Temperature Scaling
Sam Hilton-Jones, Timothy J. Norman, Zhanxing Zhu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Aitchison distance instead of KL divergence to compare LongLLaMA-3B with OpenLLaMA-3B | attention probability monitoring · LongLLaMA-3B vs OpenLLaMA-3B, memory layer 6 | direction of uniformity change | Improved |
| scale logits with log L or (log L)^2 inverse temperature instead of sqrt(log L) or none (ablation) | Pythia-1B pretraining on RedPajama sample · Pythia-1B, RedPajama-Data-1T-Sample 1B | training stability / loss | Hurt |
| use higher-order inverse-temperature function of context length (ablation) | Pythia-1B attention · Pythia-1B, Row 2048 vs Row 8 | normalised Aitchison distance gap between long and short rows | Improved |
| use aggressive training (high learning rate, no weight decay) instead of standard hyperparameters (ablation) | Pythia-1B with sqrt(log L) scaling · Pythia-1B, RedPajama sample | training loss / attention saturation | Hurt |
| use small learning rate and high weight decay instead of standard hyperparameters (ablation) | Pythia-1B with sqrt(log L) scaling · Pythia-1B, RedPajama sample | training loss | Hurt |
Closest papers by meaning
- Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calibration (ICML 2024 · similarity 0.88)
- Unifying Attention Heads and Task Vectors via Hidden State Geometry in In-Context Learning (NeurIPS 2025 · similarity 0.88)
- Selective Attention: Enhancing Transformer through Principled Context Control (NeurIPS 2024 · similarity 0.88)
- Scale-invariant attention (NeurIPS 2025 · similarity 0.88)
- A Phase Transition between Positional and Semantic Learning in a Solvable Model of Dot-Product Attention (NeurIPS 2024 · similarity 0.88)
- Dual Dimensionality for Local and Global Attention (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
