Research map / Architectures and efficiency / Attention and transformers
SAS: Simulated Attention Score
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Simulated Attention Score (SAS), which uses small MLP/conv maps along the head and feature dimensions to simulate more heads and larger per-head dimensions for Q/K (and more heads for V) at near-constant parameter count, plus a parameter-efficient group-averaged aggregation (PEAA).
Problem
Standard attention variants (MHA, MQA, GQA, MLA, TPA) trade head count and per-head dimension against parameters and memory; the paper observes that more heads help when per-head size stays large, and asks how to get that benefit cheaply.
Approach
Expand the head dimension and the Q/K feature dimension with residual ReLU-MLP or convolution mappings applied to a compact head representation, simulating a larger attention model, and aggregate per-head outputs by averaging over head groups instead of a full output projection (PEAA) to keep parameters down.
Key results
- On Arxiv at training length 512 and 1024, SAS reaches perplexity 5.65 and 4.76 versus 5.82 and 4.93 for MHA.
- With 1.5B parameters and 10B tokens, SAS averages 55.69 on the seven downstream tasks versus 53.86 for TPA and 52.86 for MHA.
- At training length 16384 on the Pile, SAS reports perplexity around 8.97 at 50K steps versus 9.94 for MHA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply SAS simulated attention head and feature expansion | 125M Transformer · Arxiv, training length 512 | perplexity: 5.82 -> 5.65 | Helped |
| apply SAS simulated attention head and feature expansion | 125M Transformer · Books3, training length 1024 | perplexity: 23.67 -> 22.14 | Helped |
| apply SAS simulated attention head and feature expansion | 350M Transformer · FineWeb-Edu, 50B tokens | average accuracy: 53.76 -> 55.07 | Helped |
| add feature simulation (ablation) | SAS-HeadOnly · Books3, training length 1024 | perplexity: 22.43 -> 22.14 | Helped |
| add head simulation (ablation) | SAS-FeatureOnly · Books3, training length 1024 | perplexity: 22.79 -> 22.14 | Helped |
| add nonlinear mapping (ablation) | SAS without nonlinear mapping · Books3, training length 1024 | perplexity: 22.43 -> 22.14 | Helped |
| increase simulated attention heads from 12 to 36 (ablation) | SAS · Books3, training length 1024 | perplexity: 22.96 -> 22.14 | Helped |
| use head-convolution kernel size 1 (ablation) | SAS · Books3, training length 1024 | perplexity: 23.67 -> 22.50 | Helped |
| increase head-convolution kernel size from 1 to 7 (ablation) | SAS · Books3, training length 1024 | perplexity | Helped |
Surprising findings
- Reported training time for SAS is substantially higher than baselines (e.g., 68.07 vs 36.20 for 125M at training length 512), which the paper acknowledges as a limitation.
Limitations
- SAS increases computation and parameter count from extra heads and wider features, as acknowledged in the limitations.
- Training time is roughly 1.5-2x that of baselines in the reported cost table.
Experimental setup
Datasets: Arxiv Books3 FineWeb-Edu Pile
Benchmarks: ARC-E ARC-C HellaSwag PIQA SciQ SocialIQA WinoGrande BoolQ OBQA MMLU
Metrics: perplexity accuracy accuracy norm
Baselines: MHA MQA GQA MLA TPA
Builds on: multi-head attention tensor product attention nanoGPT lm-evaluation-harness
Closest papers by meaning
- SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking (NeurIPS 2026 · similarity 0.88)
- MoH: Multi-Head Attention as Mixture-of-Head Attention (ICML 2025 · similarity 0.88)
- Degrees of Freedom for Linear Attention: Distilling Softmax Attention with Optimal Feature Efficiency (NeurIPS 2025 · similarity 0.87)
- Token Statistics Transformer: Linear-Time Attention via Variational Rate Reduction (ICLR 2025 · similarity 0.86)
- Theory, Analysis, and Best Practices for Sigmoid Self-Attention (ICLR 2025 · similarity 0.86)
- Improving Transformers with Dynamically Composable Multi-Head Attention (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.