Research map / Architectures and efficiency / Attention and transformers
From Layers to States: A State Space Model Perspective to Deep Neural Network Layer Dynamics
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Treats layer outputs of deep CNNs and ViTs as states of a continuous process and aggregates them with a selective state space model (S6) module, S6LA, improving ImageNet classification and COCO detection.
Problem
Layer aggregation methods treat layer outputs as discrete states, which may not suit very deep networks; the paper asks whether SSM-based continuous-state aggregation can improve representational power of CNNs and ViTs.
Approach
Views the outputs of network layers as states of a continuous process and aggregates them with a Mamba-style selective SSM (S6LA), where a recurrent hidden state h is concatenated/multiplied with layer features and updated with input-dependent Delta and B.
Key results
- ResNet-50 with S6LA reaches 78.0% Top-1 on ImageNet-1K vs 76.1% vanilla and 77.5% for MRLA, at 25.8M params and 4.4B FLOPs.
- DeiT-Ti with S6LA reaches 73.3% Top-1 vs 72.6% vanilla on ImageNet-1K.
- Mask R-CNN with ResNet-50 + S6LA gets 40.6 AP^bb vs 37.2 vanilla and 40.4 for MRLA on COCO 2017.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add S6LA to ResNet-50 | ResNet-50 · ImageNet-1K val | Top-1 accuracy: 77.5 -> 78.0 | Helped |
| Add S6LA to ResNet-101 | ResNet-101 · ImageNet-1K val | Top-1 accuracy: 78.7 -> 79.1 | Helped |
| Add S6LA to ResNet-152 | ResNet-152 · ImageNet-1K val | Top-1 accuracy: 79.1 -> 79.4 | Helped |
| Add S6LA to DeiT-Ti, DeiT-S, and DeiT-B | DeiT · ImageNet-1K val | Top-1 accuracy: 73.0 -> 73.3; 80.7 -> 81.3; 82.9 -> 83.3 | Helped |
| Add S6LA to Swin-T, Swin-S, and Swin-B | Swin Transformer · ImageNet-1K val | Top-1 accuracy: 80.9 -> 81.5; 82.5 -> 83.3; 82.9 -> 83.5 | Helped |
| Add S6LA to PVTv2 backbones | PVTv2 · ImageNet-1K val | Top-1 accuracy: 70.6 -> 70.8; 78.9 -> 78.8; 81.6 -> 82.3 | Mixed |
| Add S6LA to ResNet-50 | Faster R-CNN, ResNet-50 · MS COCO 2017 | APbb: 40.1 -> 40.3 | Helped |
| Add S6LA to ResNet-50 | Mask R-CNN, ResNet-50 · MS COCO 2017 | APbb: 40.4 -> 40.6 | Helped |
| Add trainable, input-influenced h (ablation) | S6LA · ImageNet-1K, ResNet-50 | Top-1 accuracy: 77.4 -> 78.0 | Helped |
| Add trainable, input-influenced h (ablation) | S6LA · ImageNet-1K, DeiT-Ti | Top-1 accuracy: 72.5 -> 73.3 | Helped |
| Add selective mechanism (ablation) | S6LA · ImageNet-1K, ResNet-50 | Top-1 accuracy: 77.3 -> 78.0 | Helped |
| Add selective mechanism (ablation) | S6LA · ImageNet-1K, DeiT-Ti | Top-1 accuracy: 72.7 -> 73.3 | Helped |
Surprising findings
- The authors report that strengthening layer interactions leads to overfitting on ResNet under the original ResNet training recipe, suggesting stronger augmentation or pretraining is needed.
Limitations
- Ablations limited to ResNet-50 and DeiT-Ti due to resource limits.
- Theoretical justification for treating layer outputs as continuous states is heuristic; the discrete-to-continuous mapping is asserted rather than proven.
- Gains on some transformer backbones are inconsistent (e.g., PVTv2-B1 S6LA 78.8 vs MRLA 78.9 Top-1).
Experimental setup
Datasets: ImageNet-1K MS COCO 2017
Benchmarks: ImageNet-1K MS COCO 2017
Metrics: Top-1 accuracy Top-5 accuracy AP^bb AP^m Params FLOPs
Baselines: SENet CBAM A2-Net ECA-Net RLA MRLA BA-Net DIANet
Builds on: Mamba S4 RLA DenseNet ResNet
Closest papers by meaning
- State Space Models are Provably Comparable to Transformers in Dynamic Token Selection (ICLR 2025 · similarity 0.88)
- On Feature Learning in Structured State Space Models (NeurIPS 2024 · similarity 0.87)
- Layer-Adaptive State Pruning for Deep State Space Models (NeurIPS 2024 · similarity 0.87)
- State Space Models for Event Cameras (CVPR 2024 · similarity 0.86)
- Theoretical Foundations of Deep Selective State-Space Models (NeurIPS 2024 · similarity 0.86)
- Understanding the Differences in Foundation Models: Attention, State Space Models, and Recurrent Neural Networks (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.