Research map / Architectures and efficiency / Attention and transformers
Pay Attention to Attention Distribution: A New Local Lipschitz Bound for Transformers
Mikalai Yudzin, Sergei Kudriashov, Alexander Gaponov, Maxim Rakhuba
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add JaSMin k=0 regularizer | ViT-B baseline · CIFAR-100 (ViT-B) | PGD4 accuracy: 13.11 -> 18.90 | Improved |
| add JaSMin k=0 regularizer | ViT-B baseline · CIFAR-100 (ViT-B) | Standard accuracy: 91.05 -> 88.83 | Hurt |
| add JaSMin k=0 regularizer | ViT-B baseline · CIFAR-100 (ViT-B) | AA2 accuracy: 2.10 -> 3.62 | Improved |
| add JaSMin k=10 lambda=1e-3 regularizer | ViT-B baseline · CIFAR-100 (ViT-B) | FGSM4 accuracy: 39.86 -> 42.34 | Improved |
| add JaSMin k=10 lambda=1e-3 regularizer | ViT-B baseline · CIFAR-100 (ViT-B) | Standard accuracy: 91.05 -> 91.08 | No clear effect |
| add Specformer with all three weights 1e-2 | ViT-B baseline · CIFAR-100 (ViT-B) | Standard accuracy: 91.05 -> 90.06 | Hurt |
| add Specformer weights (0,0,1e-2) | ViT-B baseline · CIFAR-100 (ViT-B) | PGD4 accuracy: 13.11 -> 12.56 | No clear effect |
| add JaSMin k=30 lambda=1e-2 | ViT-B baseline · Imagenette (ViT-B) | PGD4 accuracy: 15.13 -> 20.67 | Improved |
| add JaSMin k=30 lambda=1e-2 | ViT-B baseline · Imagenette (ViT-B) | Standard accuracy: 98.00 -> 97.07 | Hurt |
| add JaSMin k=70 lambda=1e-2 | ViT-B baseline · Imagenette (ViT-B) | AA2 accuracy | Improved |
| add Specformer (1e-2,0,0) | ViT-B baseline · Imagenette (ViT-B) | AA2 accuracy | No clear effect |
| use new bound instead of Gershgorin-based estimate for softmax Jacobian norm | softmax Jacobian estimate · ViT-L ImageNet | mean relative error: 20.19 -> 0.035 | Improved |
Closest papers by meaning
- How Smooth Is Attention? (ICML 2024 · similarity 0.90)
- On the Role of Attention Masks and LayerNorm in Transformers (NeurIPS 2024 · similarity 0.88)
- Spectral Conditioning of Attention Improves Transformer Performance (NeurIPS 2025 · similarity 0.88)
- Transformers Learn Faster with Semantic Focus (NeurIPS 2025 · similarity 0.87)
- kNN Attention Demystified: A Theoretical Exploration for Scalable Transformers (ICLR 2025 · similarity 0.87)
- Are Transformers with One Layer Self-Attention Using Low-Rank Weight Matrices Universal Approximators? (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
