Research map / Video / Video segmentation and tracking
Exploiting Temporal State Space Sharing for Video Semantic Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TV3S, a Mamba-based state space model architecture that shares temporal state across video frames with patch-parallel processing and shifted windows for video semantic segmentation.
Problem
Video semantic segmentation needs long-range temporal context while avoiding the redundant computation, high memory cost, and sequential bottlenecks of frame-by-frame, short-window, RNN, and transformer methods.
Approach
Applies Mamba S6 state space recurrence independently to each spatial patch of encoder features, carrying hidden states across frames, with a shifted-window second TSS module to exchange information across patch boundaries; inference reuses stored hidden states for arbitrarily long context.
Key results
- On VSPW with MiT-B1, TV3S reaches 40.0 mIoU, 90.7 mVC8 and 87.0 mVC16, versus 38.9 mIoU for MRCFA.
- With Swin-S backbone, TV3S reaches 50.6 mIoU on VSPW, over 8 mIoU above Mask2Former (42.1).
- On Cityscapes with MiT-B1, TV3S achieves 75.6 mIoU versus 75.1 for CFFM and MRCFA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace MRCFA with TV3S | MiT-B1 on VSPW · VSPW | mIoU: 38.9 -> 40.0 | Helped |
| replace MRCFA with TV3S | MiT-B2 on VSPW · VSPW | mIoU: 45.3 -> 46.3 | Helped |
| replace MRCFA with TV3S | MiT-B5 on VSPW · VSPW | mIoU: 49.9 -> 49.8 | Hurt |
| replace Mask2Former with TV3S | Swin-T on VSPW · VSPW | mIoU: 41.2 -> 44.9 | Helped |
| replace Mask2Former with TV3S | Swin-S on VSPW · VSPW | mIoU: 42.1 -> 50.6 | Helped |
| replace CFFM with TV3S | MiT-B1 on Cityscapes · Cityscapes | mIoU: 75.1 -> 75.6 | Helped |
| add shifted representation (ablation) | TV3S on VSPW · VSPW | mIoU: 38.9 -> 39.5 | Helped |
| add a second TSS layer (ablation) | TV3S on VSPW · VSPW | mIoU: 37.6 -> 38.9 | Helped |
| add feature maps before final prediction (ablation) | TV3S on VSPW · VSPW | mIoU: 39.5 -> 38.5 | Hurt |
| concatenate feature maps before final prediction (ablation) | TV3S on VSPW · VSPW | mIoU: 39.5 -> 39.3 | Hurt |
| increase window size from 4 to 20 (ablation) | TV3S on VSPW · VSPW | mIoU: 38.5 -> 40.0 | Helped |
| increase TV3S blocks from 1 to 4 (ablation) | TV3S on VSPW · VSPW | mIoU: 36.2 -> 40.0 | Helped |
Surprising findings
- Adding intermediate spatial features back via addition or concatenation before prediction reduced performance, which the authors attribute to disrupted temporal coherence.
Limitations
- Gains over some baselines are modest on Cityscapes (75.6 vs 75.1 mIoU).
- Temporal context gains plateau beyond about 8 to 16 frames.
- Some reported FPS values are lower than competing methods (e.g., Segformer MiT-B1 58.7 FPS vs TV3S 24.7).
Experimental setup
Datasets: VSPW Cityscapes
Benchmarks: VSPW Cityscapes
Metrics: mIoU mVC8 mVC16 GFLOPs Params FPS
Baselines: SegFormer CFFM MRCFA Mask2Former MPVSS DeepLabv3+ UperNet PSPNet OCRNet TCB ETC
Builds on: Mamba SegFormer Swin Transformer VSPW benchmark protocol
Closest papers by meaning
- Re-thinking Temporal Search for Long-Form Video Understanding (CVPR 2025 · similarity 0.88)
- VidSeg: Training-free Video Semantic Segmentation based on Diffusion Models (CVPR 2025 · similarity 0.88)
- Learning Spatial-Semantic Features for Robust Video Object Segmentation (ICLR 2025 · similarity 0.87)
- The Devil is in Temporal Token: High Quality Video Reasoning Segmentation (CVPR 2025 · similarity 0.87)
- VideoRefer Suite: Advancing Spatial-Temporal Object Understanding with Video LLM (CVPR 2025 · similarity 0.87)
- Spatial-Mamba: Effective Visual State Space Models via Structure-Aware State Fusion (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.