Research map / Architectures and efficiency / Attention and transformers
Towards Understanding the Word Sensitivity of Attention Layers: A Study via Random Features
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
The paper defines word sensitivity and shows random attention features (softmax) keep constant sensitivity to a single-word change while random features decay as 1/sqrt(n), leading to generalization lower bounds for RF but not RAF.
Problem
Whether attention layers have higher sensitivity to a single word change than fully connected random features, and how this affects the ability of models to distinguish contexts differing in one word in a long sentence.
Approach
Defines word sensitivity (WS) as the worst-case relative change of a feature map when one token embedding is perturbed; proves RF maps have WS O(1/sqrt n) while softmax random attention features have WS Omega(1) via a probabilistic-method argument that exploits softmax concentration, and links WS to generalization of fine-tuned/retrained GLMs.
Key results
- RF word sensitivity decays as O(1/sqrt n) with context length, numerically confirmed across embedding dimensions d.
- RAF (softmax attention) word sensitivity remains above 1 even for large n, while ReLU-RAF sensitivity is smaller and decreases with n.
- Theory shows fine-tuned and retrained RF models have test error lower bounded by (2-gamma)^2, while experiments show RAF models can reduce error on the perturbed sample.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use softmax attention rather than a fully connected random-feature map | random feature maps on generic normalized contexts · Generic normalized context | word sensitivity: \bigO{1/\sqrt n} = o(1) -> \Omega(1) | Helped |
| replace softmax with ReLU and renormalize attention scores (ablation) | RAF · Generic normalized context | word sensitivity | Hurt |
| increase the depth of a random-feature architecture (ablation) | DRF · Generic normalized context | word sensitivity: \bigO{\frac{e^{CL}}{\sqrt n}} | No clear effect |
| fine-tune the model on the original sample | RF · Opposite-label single-word modification | test error: > (2 - \gamma)^2 - \bigO{1/\sqrt n} | Hurt |
| add the original sample to the training set and retrain | RF · Opposite-label single-word modification | test error: > (2 - \gamma)^2 - \bigO{1/\sqrt n} | Hurt |
| fine-tune on the original sample | RAF · IMDb BERT-Base embeddings, n ∈ {40, 120} | test error | Helped |
| replace softmax with ReLU and renormalize attention scores (ablation) | RAF fine-tuning · IMDb BERT-Base embeddings, n = 120 | test error | Hurt |
| fine-tune on the original sample | RF · IMDb BERT-Base embeddings, n = 40 | test error | Mixed |
| add the original sample to the training set and retrain | RAF · IMDb BERT-Base embeddings | test error | Helped |
Surprising findings
- ReLU-RAF often violates the RF-style lower bound, though the model still cannot reach small error unless gamma is large.
Limitations
- Analysis is for a single-head attention layer with re-parameterized random weights W = W_Q^T W_K and no values matrix; the RAF result is non-constructive.
- Perturbations are taken in continuous embedding space rather than discrete tokens.
- The RAF generalization claim is supported only by experiments, not a rigorous proof for fine-tuned or retrained RAF models.
Experimental setup
Datasets: IMDB reviews
Benchmarks: IMDB
Metrics: word sensitivity relative change of feature map test error on perturbed sample
Baselines: random features (RF) ReLU-RAF deep random features (DRF)
Builds on: random features random attention features neural tangent kernel
Closest papers by meaning
- Attention with Trained Embeddings Provably Selects Important Tokens (NeurIPS 2025 · similarity 0.87)
- From Attention to Activation: Unraveling the Enigmas of Large Language Models (ICLR 2025 · similarity 0.87)
- Adaptive Layer Sparsity for Large Language Models via Activation Correlation Assessment (NeurIPS 2024 · similarity 0.86)
- SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking (NeurIPS 2026 · similarity 0.86)
- Punctuation-aware Hybrid Trainable Sparse Attention for Large Language Models (NeurIPS 2026 · similarity 0.86)
- The Closeness of In-Context Learning and Weight Shifting for Softmax Regression (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.