Research map / Point clouds / Point cloud classification
Adapting Pre-trained 3D Models for Point Cloud Video Understanding via Cross-frame Spatio-temporal Perception
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adapts frozen static point cloud pre-trained transformers (Point-BERT, Point-MAE, PointGPT-S) to point cloud video understanding via point tube embeddings and a cross-frame KNN graph adapter (GCTA), evaluated on MSR-Action3D and SHREC'17.
Problem
Static 3D point cloud pre-trained models capture spatial but not temporal dynamics, so they transfer poorly to 4D point cloud video understanding, where 4D pre-training data is scarce and costly.
Approach
Keep the static pre-trained transformer frozen and add a Point Tube Adapter embedding (spatio-temporal displacements in point tubes) plus a Geometric Constraint Temporal Adapter that builds a directed cross-frame KNN graph over point tubes (EdgeConv-style) to capture long-term temporal dynamics, fused with per-layer position encoders.
Key results
- On MSR-Action3D with 24 frames, PointGPT-S+CSA / Point-MAE+CSA reach 94.42% / 95.12%, exceeding prior self-supervised MaST-Pre (94.08%).
- On SHREC'17 gesture recognition, PointGPT-S+CSA achieves 96.5% accuracy, above the best listed baseline PointCMP (93.3%).
- Ablation on Point-MAE shows the baseline frozen model at 77.35%, rising to 94.08% with Point Tube Adapter plus GCTA and 95.12% with position encoders.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Point Tube Adapter to the frozen Point-MAE baseline (ablation) | Point-MAE on MSR-Action3D · MSR-Action3D | accuracy: 77.35 -> 83.97 | Helped |
| add Position Encoders to the Point Tube Adapter variant (ablation) | Point-MAE on MSR-Action3D · MSR-Action3D | accuracy: 83.97 -> 87.11 | Helped |
| add GCTA to the Point Tube Adapter variant (ablation) | Point-MAE on MSR-Action3D · MSR-Action3D | accuracy: 83.97 -> 94.08 | Helped |
| add Position Encoders to the Point Tube Adapter and GCTA variant (ablation) | Point-MAE on MSR-Action3D · MSR-Action3D | accuracy: 94.08 -> 95.12 | Helped |
| increase GCTA neighborhood size from k=5 to k=20 (ablation) | Point-MAE+CSA on MSR-Action3D · MSR-Action3D | accuracy: 91.99% -> 95.12% | Helped |
| increase GCTA neighborhood size beyond k=20 (ablation) | Point-MAE+CSA on MSR-Action3D · MSR-Action3D | accuracy: 1.04% and 0.69% | Hurt |
| expand GCTA temporal slice range from 4 to the full clip (ablation) | Point-MAE+CSA on MSR-Action3D · MSR-Action3D | accuracy: 93.38 -> 95.12 | Helped |
| apply CSA instead of the strongest listed baseline | Point-MAE on MSR-Action3D, 8 frames · MSR-Action3D, 8 frames | accuracy: 89.56 -> 91.41 | Helped |
| apply CSA instead of the strongest listed baseline | PointGPT-S on MSR-Action3D, 12 frames · MSR-Action3D, 12 frames | accuracy: 91.58 -> 93.08 | Helped |
| apply CSA instead of the strongest listed baseline | PointGPT-S on MSR-Action3D, 16 frames · MSR-Action3D, 16 frames | accuracy: 92.26 -> 94.08 | Helped |
| apply CSA instead of the strongest listed baseline | Point-MAE on MSR-Action3D, 24 frames · MSR-Action3D, 24 frames | accuracy: 94.76 -> 95.12 | Helped |
| apply CSA instead of the strongest listed baseline | PointGPT-S on SHREC’17 · SHREC’17 | accuracy: 95.2 -> 96.5 | Helped |
Surprising findings
- Simply freezing a static point cloud model and adding only a point tube embedding plus head gives a large jump (77.35% to 83.97%), suggesting spatial-only pre-training transfers well to video once temporal embedding is added.
Limitations
- Evaluated only on two small datasets (MSR-Action3D and SHREC'17) for action and gesture recognition.
- Performance is sensitive to the neighborhood size k and the GCTA placement, requiring tuning.
- Results on 8/12/16-frame settings are not uniformly best, e.g. Point-MAE+CSA trails in some configurations.
Experimental setup
Datasets: MSR-Action3D SHREC'17 ShapeNet
Benchmarks: MSR-Action3D SHREC'17
Metrics: accuracy
Baselines: MeteorNet PSTNet P4Transformer Kinet PPTr LeaF PST-Transformer X4D-SceneFormer C2P PointCMP PointCPSC MaST-Pre
Builds on: Point-MAE Point-BERT PointGPT PSTNet EdgeConv
Closest papers by meaning
- 3DInAction: Understanding Human Actions in 3D Point Clouds (CVPR 2024 · similarity 0.90)
- X4D-SceneFormer: Enhanced Scene Understanding on 4D Point Cloud Videos through Cross-Modal Knowledge Transfer (AAAI 2024 · similarity 0.89)
- PvNeXt: Rethinking Network Design and Temporal Motion for Point Cloud Video Recognition (ICLR 2025 · similarity 0.88)
- PMA: Towards Parameter-Efficient Point Cloud Understanding via Point Mamba Adapter (CVPR 2025 · similarity 0.87)
- Mamba4D: Efficient 4D Point Cloud Video Understanding with Disentangled Spatial-Temporal State Space Models (CVPR 2025 · similarity 0.86)
- Sonata: Self-Supervised Learning of Reliable Point Representations (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.