Research map / Architectures and efficiency / Attention and transformers
Understanding Self-attention Mechanism via Dynamical System Perspective
Zhongzhan Huang, Mingfu Liang, Jinghui Qin, Shanshan Zhong, Liang Lin
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| feed both x_t and x_{t+1} to the attention adaptor instead of only x_t (ablation) | StepNet · CIFAR100 (ResNet164) | top-1 accuracy (%): 76.34 -> 77.04 | Improved |
| feed both x_t and x_{t+1} instead of only x_{t+1} (ablation) | StepNet · CIFAR100 (ResNet164) | top-1 accuracy (%): 76.53 -> 77.04 | Improved |
| use F(x_{t+1}, x_t) instead of F(x_{t+1} - x_t) for finer stiffness estimation (ablation) | StepNet · CIFAR10 (ResNet164) | top-1 accuracy (%): 94.69 -> 95.14 | Improved |
| use the StepNet adaptor structure instead of alternative adaptors a-d (ablation) | StepNet · CIFAR100 (ResNet164) | top-1 accuracy (%): best alternative 76.83 (adaptor d) -> 77.04 | Improved |
| add StepNet attention to ResNet164 | ResNet164 · CIFAR100 | top-1 accuracy (%): 74.30 -> 77.04 | Improved |
| add StepNet to Faster R-CNN ResNet50 | Faster R-CNN ResNet50 · MS COCO | AP: 36.9 -> 39.8 | Improved |
| add StepNet to Faster R-CNN ResNet101 | Faster R-CNN ResNet101 · MS COCO | AP: 38.9 -> 41.9 | Improved |
| use StepNet instead of FCA on ResNet34 ImageNet | ResNet34 · ImageNet | top-1 accuracy (%): 75.08 -> 75.01 | No clear effect |
| higher Total Neural Stiffness (stiffness measurement ability) of the network | residual networks with attention · CIFAR100 and STL10 | accuracy correlation with TNS | Improved |
Closest papers by meaning
- Recurrent Self-Attention Dynamics: An Energy-Agnostic Perspective from Jacobians (NeurIPS 2025 · similarity 0.86)
- Dissecting the Interplay of Attention Paths in a Statistical Mechanics Theory of Transformers (NeurIPS 2024 · similarity 0.85)
- Hierarchical Self-Attention: Generalizing Neural Attention Mechanics to Multi-Scale Problems (NeurIPS 2025 · similarity 0.85)
- Understanding the Differences in Foundation Models: Attention, State Space Models, and Recurrent Neural Networks (NeurIPS 2024 · similarity 0.85)
- On the Role of Attention Masks and LayerNorm in Transformers (NeurIPS 2024 · similarity 0.84)
- High-Dimensional Learning Dynamics of Attention-Indexed Models (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
