AI Paper Atlas

Research map / Architectures and efficiency / Attention and transformers

S4ND: Modeling Images and Videos as Multidimensional Signals with State Spaces

Eric Nguyen, Karan Goel, Albert Gu, Gordon Downs, Preey Shah, Tri Dao, Stephen Baccus, Christopher Ré

NeurIPS 2022 · Architectures and efficiency › Attention and transformers · approach: Token merging and masked pretraining

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Replace self-attention with S4ND layers and drop positional encodingsViT-B · ImageNet-1ktop-1 accuracy: 78.9 -> 80.4Improved
Replace local Conv2D with S4ND layersConvNeXt-T · ImageNet-1ktop-1 accuracy: 82.1 -> 82.2No clear effect
Replace Conv2D with S4ND in isotropic backboneConv2D-ISO · CIFAR-10top-1 accuracy: 93.7 -> 94.1Improved
Replace Conv2D with S4ND in small ConvNeXtConvNeXt-M · Celeb-Atop-1 accuracy: 91.0 -> 91.3Improved
Use temporal S4 kernel instead of I3D inflationConvNeXt-I3D · HMDB-51 RGBtop-1 accuracy: 58.1 -> 62.1Improved
Use separable S3D kernels instead of I3D inflationConvNeXt · HMDB-51 RGBtop-1 accuracy: 58.1 -> 58.6No clear effect
Shorten initial temporal kernel length from 20.0 to 1.0 (ablation)S4ND-ConvNeXt-3D · HMDB-51 RGBtop-1 accuracy: 53.74 -> 62.07Improved
S4ND versus Conv2D, trained on mid resolution and tested on baseS4ND · CIFAR-10accuracy: 73.1 -> 88.77Improved
S4ND versus Conv2D, trained on low resolution and tested on baseS4ND · CIFAR-10accuracy: 33.1 -> 73.71Improved
S4ND versus Conv2D, trained on mid and tested on baseS4ND · Celeb-Aaccuracy: 80.52 -> 90.14Improved
S4ND versus FlexNet-16, trained on mid and tested on baseS4ND · CIFAR-10accuracy: 82.7 -> 88.77Improved
Progressive resizing low->base (80-20 epochs) instead of training only at baseS4ND · CIFAR-10accuracy and speed: 93.40 -> 92.32Mixed

Later papers that compared with this method

1 later accepted paper reports S4ND as a baseline in their results tables; 1 of them report a better number on most of the results they share with it.

Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy