Research map / Architectures and efficiency / Attention and transformers
Transformers Learn Low Sensitivity Functions: Investigations and Implications
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes token-wise input sensitivity as a unified inductive-bias metric showing transformers learn lower-sensitivity functions than MLPs, CNNs, ConvMixers and LSTMs, and links it to robustness, flat minima, and grokking progress.
Problem
Understanding what inductive biases distinguish transformers from other architectures, across modalities beyond Boolean inputs, and how those biases relate to robustness, generalization and training dynamics.
Approach
Extends Boolean-cube sensitivity to real-valued data by measuring output changes when individual tokens are perturbed with isotropic Gaussian noise, and uses it as a unified simplicity measure; supports it with a weak spectral bias result for linear-attention transformers in the NTK/CK regime.
Key results
- At end of training on CIFAR-10, sensitivity is 0.0050 for ViT-small and 0.0014 for ViT-simple versus 0.0827 for ResNet-18 and 0.3673 for DenseNet-121.
- On QQP, sensitivity is 0.09 for LSTM, 0.03 for RoBERTa-softmax and 0.02 for RoBERTa-ReLU.
- Adding a sensitivity regularization term to ViT-small reduces ShOp sharpness from 39.166 to 9.025 on CIFAR-10.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use transformer (ViT) architecture instead of CNNs/MLPs/ConvMixer | image classification · CIFAR-10, Fashion-MNIST, SVHN | sensitivity (lower is better) | Helped |
| Replace token-wise noise with random full-input noise for sensitivity measurement (ablation) | ViT-small vs ResNet-18 on CIFAR-10 · CIFAR-10 | sensitivity gap between architectures: ResNet-18 0.0172 vs ViT-small 0.0082 (random) vs 0.0827 vs 0.0050 (token-wise) | Mixed |
| Add sensitivity regularization (augmentation with Gaussian noise or MSE on noisy patch) during ViT training (ablation) | ViT-small · CIFAR-10 flatness (ShOp, ShPred) | sharpness (lower is flatter): ShOp 39.166 -> 9.025; ShPred 0.5346 -> 0.3982 | Helped |
| Use RoBERTa with ReLU attention instead of softmax attention (ablation) | RoBERTa language model · QQP training sensitivity | sensitivity: 0.03 -> 0.02 | Helped |
| Use RoBERTa with ReLU attention instead of softmax attention (ablation) | RoBERTa language model · MRPC training sensitivity | sensitivity: 0.002 -> 0.001 | Helped |
| Use RoBERTa (transformer) instead of LSTM | language classification · MRPC and QQP training sensitivity | sensitivity: MRPC 0.15 -> 0.002; QQP 0.09 -> 0.03 | Helped |
| Use transformer with sensitivity-based progress measure on modular addition grokking | one-layer Transformer on a+b mod 113 · grokking progress measure | sensitivity decreases from epoch 3000 while test accuracy flat | Helped |
| Train ViT with Adam instead of SGD (ablation) | ViT-small on CIFAR-10 · CIFAR-10 sensitivity | sensitivity | No clear effect |
| Vary ViT depth and number of heads (ablation) | ViT-small on CIFAR-10 · CIFAR-10 sensitivity | sensitivity | No clear effect |
| Use RoBERTa depth 4 vs 8 or 12 layers (ablation) | RoBERTa · QQP sensitivity | sensitivity | No clear effect |
Surprising findings
- ReLU attention in RoBERTa gives lower sensitivity than softmax attention, attributed to softmax sparsity increasing sensitivity to individual tokens.
- Random Gaussian noise across the whole input gives a much smaller gap between ResNet-18 and ViT-small than token-wise noise, so the token-wise metric is needed to separate architectures.
Limitations
- Theoretical spectral bias result holds only for linear attention in the infinite-width NTK/CK regime and Boolean inputs.
- Sensitivity is a correlational/prescriptive link to robustness; the paper notes the connection to flatness may not always hold in general.
- Language experiments use models trained from scratch without pretraining, limiting generality to large pretrained LLMs.
Experimental setup
Datasets: CIFAR-10 ImageNet-1k Fashion-MNIST SVHN MNIST MRPC QQP CIFAR-10-C synthetic sentiment-style vocabulary dataset modular addition
Benchmarks: CIFAR-10-C CIFAR-10 ImageNet-1k Fashion-MNIST SVHN MRPC QQP MNIST
Metrics: sensitivity test accuracy corruption accuracy sharpness (ShOp, ShPred) train accuracy
Baselines: ResNet-18 DenseNet-121 ConvMixer MLP LSTM ConvNext GPT-2
Builds on: spectral bias neural tangent kernel Boolean sensitivity grokking
Closest papers by meaning
- GaLileo: General Linear Relaxation Framework for Tightening Robustness Certification of Transformers (AAAI 2024 · similarity 0.87)
- Unraveling the Gradient Descent Dynamics of Transformers (NeurIPS 2024 · similarity 0.87)
- Transformer Layers as Painters (AAAI 2025 · similarity 0.87)
- Viewing Transformers Through the Lens of Long Convolutions Layers (ICML 2024 · similarity 0.87)
- Adventures of Trustworthy Vision-Language Models: A Survey (AAAI 2024 · similarity 0.87)
- The Quantization Benefits of Residual-Free Transformers (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.