Research map / Architectures and efficiency / Attention and transformers
GSPN-2: Efficient Parallel Sequence Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Redesigns GSPN's line-scan propagation into a single fused CUDA kernel with coalesced memory, channel-shared weights, and a compressive channel proxy, achieving large speedups over GSPN-1 and competitive ImageNet and text-to-image results.
Problem
GSPN-1's line-scan spatial propagation is theoretically near-linear but its CUDA implementation launches thousands of tiny kernels, reuses HBM poorly, and saturates GPU block scheduling as channels grow, so it fails to deliver practical speedups for high-resolution vision.
Approach
Fuse all column steps of the line scan into one CUDA kernel with a (chunk, batch, channel) grid, coalesced accesses and optional shared-memory caching of hidden states, plus channel-shared propagation weights and a low-dimensional channel proxy (C_proxy) projected back by a 1x1 convolution to reduce concurrent block count.
Key results
- On an A100, a 1024x1024x8 input forward pass drops from 71.4 ms (GSPN-1) to 1.8 ms (GSPN-2), a 40.0x speedup in the step-by-step profile.
- GSPN-2-T reaches 83.0% ImageNet-1K top-1 with 24M params and 4.2G MACs, matching GSPN-T accuracy with fewer params and MACs; GSPN-2-B reaches 84.9% vs GSPN-B 84.3%.
- On COCO 1024x1024 generation, GSPN-2 gets FID 33.21 and CLIP-T 0.286, close to SD-v1.5 (FID 32.71, CLIP-T 0.290) but slightly behind GSPN-1 on those metrics.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace GSPN-1 with GSPN-2 | GSPN on ImageNet-1K, Tiny · ImageNet-1K, 224², Tiny | accuracy: 82.7 -> 83.0 | Helped |
| Replace GSPN-1 with GSPN-2 | GSPN on ImageNet-1K, Small · ImageNet-1K, 224², Small | accuracy: 83.8 -> 84.4 | Helped |
| Replace GSPN-1 with GSPN-2 | GSPN on ImageNet-1K, Base · ImageNet-1K, 224², Base | accuracy: 84.3 -> 84.9 | Helped |
| Use GSPN-2 instead of GSPN-1 | GSPN, A100 1024×1024 batch 16, 8 channels · A100, 1024×1024, batch 16, 8 channels | forward runtime: 71.4 ms -> 1.8 ms | Helped |
| Use GSPN-2 instead of GSPN-1 | GSPN, A100 1024×1024 batch 1, 1152 channels · A100, 1024×1024, batch 1, 1152 channels | forward runtime: 863.2 ms -> 5.7 ms | Helped |
| Use GSPN-2 instead of GSPN-1 | GSPN, COCO text-to-image · COCO, 1024×1024 text-to-image | FID: GSPN-2: 33.21; SD-v1.5: 32.71 | Hurt |
| Use GSPN-2 instead of GSPN-1 | GSPN, COCO text-to-image · COCO, 1024×1024 text-to-image | CLIP-T: GSPN-2: 0.286; SD-v1.5: 0.290 | Hurt |
| Fuse propagation steps into one kernel (ablation) | GSPN CUDA kernel, batch 16, 8 channels · A100, 1024×1024, batch 16, 8 channels | forward runtime: 71.4 ms -> 57.4 ms; 1.2× | Helped |
| Coalesce global-memory access (ablation) | GSPN CUDA kernel, batch 16, 8 channels · A100, 1024×1024, batch 16, 8 channels | forward runtime: 57.4 ms -> 2.4 ms; 23.9× | Helped |
| Add shared-memory caching of hidden states (ablation) | GSPN CUDA kernel, batch 16, 8 channels · A100, 1024×1024, batch 16, 8 channels | forward runtime: 2.4 ms -> 2.2 ms; 1.1× | Helped |
| Use 2D thread blocks (ablation) | GSPN CUDA kernel, batch 16, 8 channels · A100, 1024×1024, batch 16, 8 channels | forward runtime: 2.2 ms -> 2.1 ms; 1.1× | Helped |
| Add compressive proxy channels (ablation) | GSPN CUDA kernel, batch 16, 8 channels · A100, 1024×1024, batch 16, 8 channels | forward runtime: 2.1 ms -> 1.9 ms; 1.1× | Helped |
Surprising findings
- Explicit shared-memory caching of hidden states gave a 0.9x slowdown in the single-channel, large-batch configuration, with L1 cache already capturing reuse.
- GSPN-1 slightly outperforms GSPN-2 on text-to-image FID and CLIP-T metrics, as the paper acknowledges.
Limitations
- Performance gains diminish when batch size times channel count is small, and SM occupancy can drop to 20-30%.
- Long-context video evaluation is underexplored, and the implementation lacks CLS/register tokens, limiting drop-in use as attention replacement.
- Dense prediction evaluation uses mainly 480-512 pixel images, so higher-resolution testing is missing.
Experimental setup
Datasets: ImageNet-1K COCO
Benchmarks: ImageNet-1K COCO
Metrics: top-1 accuracy forward/backward runtime (ms) global memory throughput (GB/s) FID CLIP-T MACs parameters throughput (img/s)
Baselines: GSPN-1 ConvNeXt Swin Transformer VMamba LocalVMamba SDXL Stable Diffusion v1.5 Mamba Mamba2 Linfusion
Builds on: GSPN Stable Diffusion XL Mamba LPU MESA
Closest papers by meaning
- Parallel Sequence Modeling via Generalized Spatial Propagation Network (CVPR 2025 · similarity 0.93)
- Recursive Generalization Transformer for Image Super-Resolution (ICLR 2024 · similarity 0.86)
- Image Processing GNN: Breaking Rigidity in Super-Resolution (CVPR 2024 · similarity 0.84)
- ViG: Linear-complexity Visual Sequence Learning with Gated Linear Attention (AAAI 2025 · similarity 0.84)
- Gaussian Graph Network: Learning Efficient and Generalizable Gaussian Representations from Multi-view Images (NeurIPS 2024 · similarity 0.84)
- MoDec-GS: Global-to-Local Motion Decomposition and Temporal Interval Adjustment for Compact Dynamic 3D Gaussian Splatting (CVPR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.