AI Paper Atlas

Research map / Architectures and efficiency / Attention and transformers

Understanding Self-attention Mechanism via Dynamical System Perspective

Zhongzhan Huang, Mingfu Liang, Jinghui Qin, Shanshan Zhong, Liang Lin

ICCV 2023 · Architectures and efficiency › Attention and transformers · approach: Sparse and structured attention · 52 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
feed both x_t and x_{t+1} to the attention adaptor instead of only x_t (ablation)StepNet · CIFAR100 (ResNet164)top-1 accuracy (%): 76.34 -> 77.04Improved
feed both x_t and x_{t+1} instead of only x_{t+1} (ablation)StepNet · CIFAR100 (ResNet164)top-1 accuracy (%): 76.53 -> 77.04Improved
use F(x_{t+1}, x_t) instead of F(x_{t+1} - x_t) for finer stiffness estimation (ablation)StepNet · CIFAR10 (ResNet164)top-1 accuracy (%): 94.69 -> 95.14Improved
use the StepNet adaptor structure instead of alternative adaptors a-d (ablation)StepNet · CIFAR100 (ResNet164)top-1 accuracy (%): best alternative 76.83 (adaptor d) -> 77.04Improved
add StepNet attention to ResNet164ResNet164 · CIFAR100top-1 accuracy (%): 74.30 -> 77.04Improved
add StepNet to Faster R-CNN ResNet50Faster R-CNN ResNet50 · MS COCOAP: 36.9 -> 39.8Improved
add StepNet to Faster R-CNN ResNet101Faster R-CNN ResNet101 · MS COCOAP: 38.9 -> 41.9Improved
use StepNet instead of FCA on ResNet34 ImageNetResNet34 · ImageNettop-1 accuracy (%): 75.08 -> 75.01No clear effect
higher Total Neural Stiffness (stiffness measurement ability) of the networkresidual networks with attention · CIFAR100 and STL10accuracy correlation with TNSImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy