Research map / Architectures and efficiency / Attention and transformers
What Makes Convolutional Models Great on Long Sequence Modeling?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SGConv instead of the well-tuned S4 kernel | Global convolution on Long Range Arena · Long Range Arena (average) | accuracy: 86.09 -> 87.17 | Helped |
| Add a decaying weight structure to the kernel (ablation) | SGConv on IMDB · IMDB byte-level document classification | accuracy | Helped |
| Increase scale dimension from 8 toward 64 (ablation) | SGConv on IMDB · IMDB byte-level document classification | accuracy | Hurt |
| Vary scale dimension within a reasonable range rather than changing decay speed (ablation) | SGConv on IMDB · IMDB byte-level document classification | accuracy | No clear effect |
| Use cosine-wave kernel initialization and randomly vary alpha across channels (ablation) | SGConv on Pathfinder-X · Pathfinder-X | accuracy: 1% | Helped |
| Use the disentangled kernel parameterization and 1/pos decay (ablation) | SGConv on LRA Image · LRA Image | accuracy: 1% | Helped |
| Replace half of Transformer attention blocks with SGConv blocks, placing SGConv deeper | Transformer-style language model · WikiText-103 | perplexity: 24.0 -> 22.83 | Helped |
| Replace ConvNeXt 7×7 kernels with SGConv | ConvNeXt-S · ImageNet-1k val | Top-1 accuracy: 83.1 -> 83.4 | Helped |
| Replace ConvNeXt 7×7 kernels with SGConv | ConvNeXt-T · ImageNet-1k val | Top-1 accuracy: 82.1 -> 82.0 | Hurt |
Closest papers by meaning
- Efficiently Modeling Long Sequences with Structured State Spaces (ICLR 2022 · similarity 0.90)
- Reparameterized Multi-Resolution Convolutions for Long Sequence Modelling (NeurIPS 2024 · similarity 0.87)
- FlashFFTConv: Efficient Convolutions for Long Sequences with Tensor Cores (ICLR 2024 · similarity 0.85)
- Short-Long Convolutions Help Hardware-Efficient Linear Attention to Focus on Long Sequences (ICML 2024 · similarity 0.85)
- Orchid: Flexible and Data-Dependent Convolution for Sequence Modeling (NeurIPS 2024 · similarity 0.84)
- Never Train from Scratch: Fair Comparison of Long-Sequence Models Requires Data-Driven Priors (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.