Research map / Point clouds / Point cloud classification
Mamba4D: Efficient 4D Point Cloud Video Understanding with Disentangled Spatial-Temporal State Space Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Mamba4D, a pure state-space-model 4D point cloud video backbone that disentangles intra-frame spatial Mamba and inter-frame temporal Mamba, achieving accuracy gains and large efficiency improvements over transformer baselines.
Problem
Building an effective and efficient backbone for 4D point cloud video understanding, where irregular spatial structure, temporal inconsistency, and the quadratic cost of transformers hinder long-sequence modeling.
Approach
Replace CNN and transformer components of 4D backbones with state space (Mamba) blocks: an intra-frame spatial Mamba with temporal pooling over point tubes for short-term structure, followed by an inter-frame temporal Mamba with multiple spatio-temporal scanning orders for long-range dependencies, giving linear complexity.
Key results
- On MSR-Action3D, PST-Transformer + Ours reaches 93.38% accuracy (24 frames), versus 93.03% for the PST-Transformer baseline.
- On HOI4D action segmentation, Ours achieves 85.5 Acc and 85.5 F1@50, surpassing X4D-SceneFormer's 84.1 Acc and 84.8 F1@50.
- Substituting P4Transformer's backbones with Mamba yields 87.5% GPU memory reduction and 5.36x inference speed-up on long sequences; Synthia 4D mIoU improves by 0.19 over P4Transformer (83.16 to 83.35).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace Point 4D convolution with Intra-frame Spatial Mamba | PST-Transformer · MSR-Action3D, 24 frames | accuracy: 93.03 -> 93.38 | Helped |
| Use Mamba4D with X4D-SceneFormer backbone | X4D-SceneFormer · HOI4D | F1@50: 84.8 -> 85.5 | Helped |
| Replace P4Transformer's CNN and transformer architectures with Mamba blocks | P4Transformer · Synthia 4D, 3 frames | mIoU: 83.16 -> 83.35 | Helped |
| Add Intra-frame Spatial Mamba and Inter-frame Temporal Mamba (ablation) | P4Transformer · MSR-Action3D | accuracy: 90.94 -> 92.68 | Helped |
| Add Intra-frame Spatial Mamba (ablation) | P4Transformer · MSR-Action3D | accuracy: 90.94 -> 91.36 | Helped |
| Add Inter-frame Temporal Mamba (ablation) | P4Transformer · MSR-Action3D | accuracy: 90.94 -> 91.67 | Helped |
| Use 4D rather than no position encoding (ablation) | Intra-frame Spatial Mamba · MSR-Action3D | accuracy: 89.20 -> 92.68 | Helped |
| Use 4D rather than 3D spatial-only position encoding (ablation) | Intra-frame Spatial Mamba · MSR-Action3D | accuracy: 91.98 -> 92.68 | Helped |
| Use 4D rather than no position encoding (ablation) | Inter-frame Temporal Mamba · MSR-Action3D | accuracy: 88.85 -> 92.68 | Helped |
| Use 4D rather than 3D spatial-only position encoding (ablation) | Inter-frame Temporal Mamba · MSR-Action3D | accuracy: 91.98 -> 92.68 | Helped |
| Use 12 rather than 4 Intra-frame Mamba blocks, with 4 Inter-frame blocks (ablation) | Mamba4D · MSR-Action3D | accuracy: 90.24 -> 92.68 | Helped |
| Use cross-temporal XZY ordering rather than unidirectional ordering (ablation) | Inter-frame Temporal Mamba · MSR-Action3D | accuracy: 91.36 -> 92.68 | Helped |
Surprising findings
- Inter-frame temporal Mamba with 12 intra-frame blocks and 4 inter-frame blocks performs worse than the 4/4 setting in the block-count ablation, suggesting deeper stacks are not always better.
Limitations
- Gains on Synthia 4D segmentation are marginal (0.19 mIoU), which the authors attribute to segmentation being less time-dependent.
- Action recognition gain of +10.4% reported in the abstract is relative to a weaker transformer baseline setting and is not uniform across all compared methods.
Experimental setup
Datasets: MSR-Action3D HOI4D Synthia 4D
Benchmarks: MSR-Action3D HOI4D Synthia 4D
Metrics: accuracy frame-level accuracy edit distance F1@10 F1@25 F1@50 mIoU GPU memory inference time
Baselines: PointNet++ MeteorNet PSTNet PPTr MaST-Pre P4Transformer PST-Transformer X4D-SceneFormer C2P 3D MinkNet14 4D MinkNet14
Builds on: Mamba P4Transformer PST-Transformer Point Mamba X4D-SceneFormer
Closest papers by meaning
- Adapting Pre-trained 3D Models for Point Cloud Video Understanding via Cross-frame Spatio-temporal Perception (CVPR 2025 · similarity 0.86)
- X4D-SceneFormer: Enhanced Scene Understanding on 4D Point Cloud Videos through Cross-Modal Knowledge Transfer (AAAI 2024 · similarity 0.85)
- 3DInAction: Understanding Human Actions in 3D Point Clouds (CVPR 2024 · similarity 0.83)
- Motion4D: Learning 3D-Consistent Motion and Semantics for 4D Scene Understanding (NeurIPS 2025 · similarity 0.83)
- PvNeXt: Rethinking Network Design and Temporal Motion for Point Cloud Video Recognition (ICLR 2025 · similarity 0.83)
- A Unified Framework for Human-centric Point Cloud Video Understanding (CVPR 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.