Research map / Point clouds / Point cloud classification
X4D-SceneFormer: Enhanced Scene Understanding on 4D Point Cloud Videos through Cross-Modal Knowledge Transfer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes X4D-SceneFormer, a training-only cross-modal knowledge transfer framework that uses RGB sequences and temporal gradients to improve a point-cloud-only 4D model for action segmentation, semantic segmentation, and action recognition.
Problem
4D point cloud video understanding is hard due to sparse, texture-less point clouds and the irregularity that complicates temporal alignment; adding RGB at inference costs extra computation.
Approach
Use RGB sequences and temporal gradients only during training, via a Gradient-aware Image Transformer and a cross-modal transformer with an attention mask that blocks point-to-image attention, plus temporal-aware consistency and contrastive losses, so the point model runs alone at inference.
Key results
- Action segmentation on HOI4D test: 85.3% accuracy and 91.5 edit, +7.9% accuracy over PPTr baseline (77.4%), reported as 1st place on the challenge leaderboard.
- Semantic segmentation on HOI4D test: 47.3% mIoU vs 41.4% for PPTr baseline (+5.0% mIoU).
- MSR-Action3D with 16 frames: 92.56% top-1 vs 90.31% for PPTr, reported as slightly below PPTr+C2P at 24 frames (94.76%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add cross-modal RGB input with simple concatenation instead of GIT (ablation) | PPTr on HOI4D action segmentation · HOI4D action segmentation validation | Acc: 72.3 -> 77.5 | Helped |
| Add TG to point-only input with simple concatenation (ablation) | PPTr on HOI4D action segmentation · HOI4D action segmentation validation | Acc: 72.3 -> 72.8 | No clear effect |
| Add TG to point and RGB inputs with simple concatenation (ablation) | PPTr on HOI4D action segmentation · HOI4D action segmentation validation | F1@10: 75.7 -> 73.6 | Hurt |
| Add correlation feature to the vanilla model (ablation) | X4D-SceneFormer on HOI4D action segmentation · HOI4D action segmentation validation | Acc: 76.8 -> 81.2 | Helped |
| Add sliding window to the correlation feature (ablation) | X4D-SceneFormer on HOI4D action segmentation · HOI4D action segmentation validation | Edit: 82.5 -> 84.5 | Helped |
| Add temporal contrastive loss (ablation) | X4D-SceneFormer on HOI4D action segmentation · HOI4D action segmentation validation | Edit: 84.5 -> 87.9 | Helped |
| Add temporal consistency loss (ablation) | X4D-SceneFormer on HOI4D action segmentation · HOI4D action segmentation validation | Edit: 87.9 -> 92.4 | Helped |
| Use cross-attention rather than concatenation, sum, or self-attention fusion (ablation) | GIT on HOI4D action segmentation · HOI4D action segmentation validation | Acc: 79.7 -> 82.6 | Helped |
| Replace proposed knowledge transfer with classic distillation alternatives (ablation) | X4D-SceneFormer on HOI4D action segmentation · HOI4D action segmentation validation | Acc: 82.6 -> 73.8 | Hurt |
| Use sliding window size 3 instead of sizes 1, 5, or 7 (ablation) | X4D-SceneFormer on HOI4D action segmentation · HOI4D action segmentation validation | Edit: 84.5 -> 92.4; 84.1 -> 92.4; 82.1 -> 92.4 | Helped |
| Use fast rather than slow or medium TG stride (ablation) | X4D-SceneFormer on HOI4D action segmentation · HOI4D action segmentation validation | Acc: 80.4 -> 82.6; 80.2 -> 82.6 | Helped |
| Add self-attention layers to PPTr without image input (ablation) | PPTr on HOI4D action segmentation · HOI4D action segmentation validation | Acc: 72.3 -> 72.4 | No clear effect |
Surprising findings
- Naively adding temporal gradient input to the cross-modal transformer degraded accuracy (72.8 vs 77.5 with RGB only), which the authors attribute to temporal inconsistency between modalities.
- Without the attention mask that blocks point-to-image attention during training, point-only inference accuracy dropped sharply (62.4 vs 82.6 with mask).
Limitations
- Naive temporal gradient input hurts performance, and gains depend on carefully designed temporal consistency losses.
- On MSR-Action3D, uses projected depth maps instead of RGB and a weaker baseline, so results are slightly below PPTr+C2P at some frame counts.
- HOI4D test set is not public, so validation split is randomly carved from training data; 2D segmentation labels are derived by projecting point labels.
Experimental setup
Datasets: HOI4D MSR-Action3D
Benchmarks: HOI4D action segmentation HOI4D semantic segmentation MSR-Action3D
Metrics: frame-wise accuracy edit score F1@{10,25,50} mIoU top-1 video accuracy
Baselines: P4Transformer PPTr PPTr+C2P Multi-Conv-Res DPMix MeteorNet PSTNet PointNet++
Builds on: P4Transformer PPTr P4Conv ResNet
Closest papers by meaning
- Motion4D: Learning 3D-Consistent Motion and Semantics for 4D Scene Understanding (NeurIPS 2025 · similarity 0.89)
- Adapting Pre-trained 3D Models for Point Cloud Video Understanding via Cross-frame Spatio-temporal Perception (CVPR 2025 · similarity 0.89)
- L4D-Track: Language-to-4D Modeling Towards 6-DoF Tracking and Shape Reconstruction in 3D Point Cloud Stream (CVPR 2024 · similarity 0.88)
- IGGT4D: Streaming 4D Instance-Grounded Geometry Transformer (NeurIPS 2026 · similarity 0.87)
- 4DP-QA: Scalable QA for 4D Perception in Vision Language Models (CVPR 2026 · similarity 0.87)
- 3DInAction: Understanding Human Actions in 3D Point Clouds (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.