Research map / Architectures and efficiency / Attention and transformers
SparX: A Sparse Cross-Layer Connection Mechanism for Hierarchical Vision Mamba and Transformer Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SparX, a sparse cross-layer connection mechanism (ganglion and normal layers with channel-wise cross-attention DMCA and cross-layer sliding window) that improves Mamba- and Transformer-based hierarchical vision backbones on ImageNet-1K, COCO, and ADE20K.
Problem
Mamba-based vision models cannot effectively aggregate and select features across layers, and existing cross-layer methods (DenseNet-style connections, spatial cross-attention) are too costly in throughput and memory for Mamba-based backbones.
Approach
Inspired by retinal ganglion cells, interleave sparse "ganglion" layers (with dynamic multi-layer channel aggregation via grouped channel cross-attention, spatially reduced) and "normal" layers, and limit ganglion inter-connections to a cross-layer sliding window of M preceding ganglion layers.
Key results
- SparX-Mamba-T reaches 83.5% top-1 on ImageNet-1K with 27M params versus VMamba-T 82.5% with 31M params.
- SparX-Swin-T improves Swin-T top-1 from 81.3% to 82.6% on ImageNet-1K.
- SparX-Mamba-T with UperNet reaches 50.0 mIoU on ADE20K (160K) versus VMamba-T 48.3.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add sparse cross-layer connections with ganglion layers and DMCA | VMamba-T · ImageNet-1K | accuracy: 82.5 -> 83.5 | Helped |
| Add sparse cross-layer connections with ganglion layers and DMCA | VMamba-T · COCO 2017 Mask R-CNN 3× | APb: 48.9 -> 50.2 | Helped |
| Add sparse cross-layer connections with ganglion layers and DMCA | VMamba-T · ADE20K S-FPN 80K | mIoU: 47.2 -> 49.5 | Helped |
| Remove the cross-layer sliding window and connect ganglion layers to all preceding layers (ablation) | SparX-Mamba-T · ImageNet-1K | accuracy: 83.5 -> 83.4 | Hurt |
| Set S=1 and remove the cross-layer sliding window (ablation) | SparX-Mamba-T · ImageNet-1K | accuracy: 83.5 -> 83.4 | No clear effect |
| Replace DMCA with feature concatenation and Conv-FFN fusion (ablation) | SparX-Mamba-T · ImageNet-1K | accuracy: 83.5 -> 82.8 | Hurt |
| Remove cross-layer grouped channel attention (ablation) | SparX-Mamba-T · ImageNet-1K | accuracy: 83.5 -> 82.7 | Hurt |
| Replace DMCA with spatial cross-attention (ablation) | SparX-Mamba-T · ImageNet-1K | accuracy: 83.5 -> 83.3 | Hurt |
| Replace all ganglion layers with normal layers, removing cross-layer connections (ablation) | SparX-Mamba-T · ImageNet-1K | accuracy: 83.5 -> 82.4 | Hurt |
| Set ganglion-layer stride S=1 (ablation) | SparX-Mamba-T · ImageNet-1K | accuracy: 83.5 -> 83.5 | No clear effect |
| Set ganglion-layer stride S=4 (ablation) | SparX-Mamba-T · ImageNet-1K | accuracy: 83.5 -> 82.9 | Hurt |
| Remove Dynamic Position Encoding (ablation) | SparX-Mamba-T · ADE20K | mIoU: 49.5 -> 49.0 | Hurt |
Surprising findings
- Making VMamba DenseNet-style lowers throughput by almost 50% and adds about 1GB GPU memory, motivating the sparse design.
- Densely connecting all ganglion layers (DGC) or S=1 gave no clear accuracy gain over the sparse design.
Limitations
- Throughput is lower than VMamba-T and ConvNeXt/Swin, and some Mamba-based speed gaps remain attributed to unoptimized SSM kernels.
- Gains on ImageNet are modest (around 1 point) relative to added design complexity.
- Vanilla SSM variant lags VMamba on segmentation (48.1 vs 48.3 mIoU).
Experimental setup
Datasets: ImageNet-1K COCO 2017 ADE20K
Benchmarks: ImageNet-1K COCO 2017 ADE20K
Metrics: top-1 accuracy AP^b AP^m mIoU FLOPs params throughput GPU memory
Baselines: VMamba LocalVMamba MSVMamba EfficientVMamba ViM2 PlainMamba ConvNeXt Swin CSWin MambaOut UniRepLKNet FcaFormer
Builds on: Mamba VMamba DenseNet Swin Transformer PVT
Closest papers by meaning
- LSNet: See Large, Focus Small (CVPR 2025 · similarity 0.86)
- MobileMamba: Lightweight Multi-Receptive Visual Mamba Network (CVPR 2025 · similarity 0.86)
- Explicitly Modeling Subcortical Vision with a Neuro-Inspired Front-End Improves CNN Robustness (NeurIPS 2025 · similarity 0.86)
- Multi-Scale VMamba: Hierarchy in Hierarchy Visual State Space Model (NeurIPS 2024 · similarity 0.85)
- MambaOut: Do We Really Need Mamba for Vision? (CVPR 2025 · similarity 0.85)
- CoreMatching: A Co-adaptive Sparse Inference Framework with Token and Neuron Pruning for Comprehensive Acceleration of Vision-Language Models (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.