Research map / Architectures and efficiency / Attention and transformers
You Need Better Attention Priors
Elon Litman, Gabe Guo
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace uniform prior with learnable GOAT prior (vs ALiBi) | 125M decoder on C4 · C4 in-distribution perplexity at L=2048 | perplexity: 32.49 -> 30.95 | Improved |
| replace RoPE with GOAT prior for length extrapolation | 1.23B Llama-style decoder on FineWeb · FineWeb perplexity at 16k (trained at 2048) | perplexity: 417.43 -> 24.77 | Improved |
| replace RoPE with GOAT at the training length | 1.23B Llama-style decoder on FineWeb · FineWeb perplexity at 2k | perplexity: 20.82 -> 20.75 | No clear effect |
| use GOAT prior instead of RoPE, position-interpolated RoPE, ALiBi or sinusoidal PE | GPT-style LM · Passkey retrieval and Needle-in-a-Haystack | span token accuracy | Improved |
| use GOAT prior instead of learnable absolute positional embedding in a ViT | ViT on ImageNet-1k · ImageNet-1k top-1, trained at 224px, tested at higher resolution | top-1 accuracy | Improved |
| use GOAT instead of RoPE on human genome sequences | DNA language model · human reference genome next-token modeling | validation NLL | Improved |
| add explicit key-only sink bias u(j) separate from content keys | GOAT · 125M C4 / synthetic copy-mixture | learned prior structure | Improved |
| use fixed ALiBi slopes versus learned prior | ALiBi · Needle-in-a-Haystack | span token accuracy | Hurt |
| use ALiBi for length extrapolation instead of RoPE | 125M decoder on C4 · C4 length extrapolation | perplexity: in-distribution 32.49 (ALiBi) vs 33.24 (RoPE) | Mixed |
Closest papers by meaning
- Transformers Learn Faster with Semantic Focus (NeurIPS 2025 · similarity 0.86)
- Sparse Attention as Compact Kernel Regression (NeurIPS 2026 · similarity 0.85)
- Bayes optimal learning of attention-indexed models (NeurIPS 2025 · similarity 0.85)
- Towards Interpretable and Efficient Attention: Compressing All by Contracting a Few (NeurIPS 2025 · similarity 0.85)
- Sink vs. diagonal patterns as mechanisms for attention switch and oversmoothing prevention (NeurIPS 2026 · similarity 0.85)
- The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicry (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
