Research map / Video / Video segmentation and tracking
State Space Prompting via Gathering and Spreading Spatio-Temporal Information for Video Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes State Space Prompting (SSP), a parameter-efficient prompt method for VideoMamba that combines intra-frame gathering and inter-frame spreading prompts to propagate spatio-temporal information through the state space compression.
Problem
Visual prompt methods designed for Transformers fail to preserve spatial and temporal context when applied to sequentially compressing state space models like VideoMamba, causing information decay in video understanding and limiting downstream adaptation.
Approach
Two complementary prompt modules: an Intra-Frame Gathering module uses low-rank convolution to produce spatial prompts plus entropy and spatial-variance gates, and an Inter-Frame Spreading module uses low-rank attention over last-token samples per frame to produce temporal prompts inserted between frames, so that the Mamba state path length between distant tokens is shortened.
Key results
- With VideoMamba-M, SSP reaches 76.66% on HMDB51, 53.72% on SSV2, 97.03% on UCF101 and 93.23% on Breakfast, improving over prior PEFT methods by 3.00, 4.64, 0.26 and 3.13 points.
- SSP uses 2.41M tunable parameters with VideoMamba-M and 0.98M with VideoMamba-S, versus 74.00M for full tuning of VideoMamba-M.
- Ablation: IFG alone and IFS alone raise average accuracy over the no-prompt baseline; using both gives the best results across HMDB51, UCF101 and Breakfast.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add SSP prompts | VideoMamba-S · HMDB51, SSV2, UCF101, Breakfast | accuracy: 74.38; 38.68; 95.69; 85.41 | Helped |
| Add SSP prompts | VideoMamba-M · HMDB51, SSV2, UCF101, Breakfast | accuracy: 76.66; 53.72; 97.03; 93.23 | Helped |
| Add the intra-frame gathering module (IFG) (ablation) | SSP · HMDB51, UCF101, Breakfast | accuracy: 59.34 -> 74.44; 90.64 -> 96.56; 76.56 -> 84.89 | Helped |
| Add the inter-frame spreading module (IFS) (ablation) | SSP · HMDB51, UCF101, Breakfast | accuracy: 59.34 -> 72.61; 90.64 -> 96.14; 76.56 -> 85.41 | Helped |
| Add both IFG and IFS (ablation) | SSP · HMDB51, UCF101, Breakfast | accuracy: 59.34 -> 76.66; 90.64 -> 97.03; 76.56 -> 93.23 | Helped |
| Add the entropy gate (ablation) | SSP · HMDB51, UCF101, Breakfast | accuracy: 74.70 -> 75.09; 96.56 -> 96.29; 89.58 -> 88.54 | Helped |
| Add the spatial variance gate (ablation) | SSP · HMDB51, UCF101, Breakfast | accuracy: 74.70 -> 74.96; 96.56 -> 96.61; 89.58 -> 88.54 | Helped |
| Sample the middle rather than the last token in each frame (ablation) | SSP inter-frame prompt generation · HMDB51, UCF101, Breakfast | accuracy: 74.44; 95.87; 91.14 | Hurt |
| Sample last tokens separately from forward and backward scans (ablation) | SSP inter-frame prompt generation · HMDB51, UCF101, Breakfast | accuracy: 75.81; 96.00; 89.06 | Hurt |
| Use independent IFS modules for forward and backward directions (ablation) | SSP · HMDB51, UCF101, Breakfast | accuracy: 75.16; 96.06; 90.10 | Hurt |
| Increase the number of IFS modules (ablation) | SSP · Video benchmarks | accuracy | Mixed |
| Increase the internal dimension of IFG (ablation) | SSP · HMDB51 and Breakfast | accuracy | Mixed |
Surprising findings
- Sampling inter-frame prompt tokens from the middle of frames, or from separate forward and backward scans, lowers accuracy relative to the last-token forward sampling used by SSP.
Limitations
- Adds hyperparameters (number of IFS modules, internal dimensions of IFG and IFS) that must be tuned; the paper recommends fixed values per backbone.
- Reports single runs without error bars, citing dataset size and cost.
Experimental setup
Datasets: Kinetics-400 HMDB51 UCF101 Something-Something V2 Breakfast
Benchmarks: HMDB51 UCF101 Something-Something V2 Breakfast
Metrics: top-1 accuracy
Baselines: Full fine-tuning Adapter ST-Adapter VPT VFPT SVP Additional-Scan STOP DGL-Linear DGL-Transformer
Builds on: VideoMamba VPT STOP SVP Additional-Scan
Closest papers by meaning
- STOP: Integrated Spatial-Temporal Dynamic Prompting for Video Understanding (CVPR 2025 · similarity 0.91)
- Learning Conditional Space-Time Prompt Distributions for Video Class-Incremental Learning (CVPR 2025 · similarity 0.88)
- SA²VP: Spatially Aligned-and-Adapted Visual Prompt (AAAI 2024 · similarity 0.86)
- ViKey: Enhancing Temporal Understanding in Videos via Visual Prompting (CVPR 2026 · similarity 0.86)
- DELTAVID: Enhancing Fine-Grained Spatiotemporal Perception with Cross-Video Differences (NeurIPS 2026 · similarity 0.86)
- Compressed Video Prompt Tuning (NeurIPS 2023 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.