Research map / Architectures and efficiency / Attention and transformers
Bayesian Attention Mechanism: A Probabilistic Framework for Positional Encoding and Context Length Extrapolation
Arthur S. Bianchessi, Yasmin C. Aguirre, Rodrigo C. Barros, Lucas S. Kupssinskü
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Scaled Softmax (SSMax) to BAM (ablation) | BAM · Wikipedia (512 tokens) | perplexity: 18.9281 -> 18.6897 | Improved |
| add Scaled Softmax (SSMax) to BAM (ablation) | BAM · FineWeb (512 tokens) | perplexity: 22.2640 -> 22.1363 | Improved |
| add Scaled Softmax (SSMax) to RoPE (ablation) | RoPE · Wikipedia (512 tokens) | perplexity: 18.3599 -> 20.1854 | Hurt |
| add Scaled Softmax (SSMax) to RoPE (ablation) | RoPE · FineWeb (512 tokens) | perplexity: 22.4428 -> 24.4507 | Hurt |
| add Scaled Softmax (SSMax) to NoPE (ablation) | NoPE · Wikipedia (512 tokens) | perplexity: 19.8609 -> 20.3899 | Hurt |
| add Scaled Softmax (SSMax) to NoPE (ablation) | NoPE · FineWeb (512 tokens) | perplexity: 23.7898 -> 23.7581 | Improved |
| add Scaled Softmax (SSMax) to ALiBi (ablation) | ALiBi · Wikipedia and FineWeb (512 tokens) | perplexity: 18.5452 -> 18.4967 (Wikipedia); 22.1771 -> 22.2125 (FineWeb) | Mixed |
| add Scaled Softmax (SSMax) to Sinusoidal (ablation) | Sinusoidal · Wikipedia and FineWeb (512 tokens) | perplexity: 18.9310 -> 19.1467 (Wikipedia); 22.3573 -> 22.3150 (FineWeb) | Mixed |
| use BAM instead of ALiBi positional prior | ALiBi · Wikipedia (512 tokens) | perplexity: 18.5452 -> 18.9281 | Hurt |
| use BAM instead of Sinusoidal | Sinusoidal · FineWeb (512 tokens) | perplexity: 22.3573 -> 22.2640 | Improved |
| use BAM instead of Sinusoidal, RoPE and NoPE for passkey retrieval beyond training length | Sinusoidal/RoPE/NoPE · Passkey retrieval | accuracy: only BAM retained high accuracy up to 32,000 tokens | Improved |
| add BAM SSMax to ALiBi baseline for passkey retrieval | ALiBi · Passkey retrieval | accuracy: perfect accuracy beyond 64x training length | Improved |
Closest papers by meaning
- Exploring Context Window of Large Language Models via Decomposed Positional Vectors (NeurIPS 2024 · similarity 0.89)
- The Role of Sparsity for Length Generalization in LLMs (ICML 2025 · similarity 0.88)
- GOLD PANNING: Strategic Context Shuffling for Needle-in-Haystack Reasoning (NeurIPS 2026 · similarity 0.88)
- Rethinking Addressing in Language Models via Contextualized Equivariant Positional Encoding (ICML 2025 · similarity 0.88)
- Eliminating Position Bias of Language Models: A Mechanistic Approach (ICLR 2025 · similarity 0.88)
- Efficient Length-Generalizable Attention via Causal Retrieval for Long-Context Language Modeling (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
