Research map / Point clouds / Point cloud classification
L4D-Track: Language-to-4D Modeling Towards 6-DoF Tracking and Shape Reconstruction in 3D Point Cloud Stream
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes L4D-Track, a language-to-4D framework that jointly performs zero-shot 6-DoF category-level pose tracking and shape reconstruction on 3D point cloud streams, trained only on NOCS-REAL275 with GPT-assisted multi-level caption contrastive alignment.
Problem
Zero-shot 6-DoF pose tracking and 3D shape reconstruction of unseen objects from 3D point cloud video, where large-scale 3D-language paired data is scarce.
Approach
Pairwise implicit 3D space representation (energy-based pose hypothesis plus neural pose-aligned fields) over consecutive point cloud frames, aligned to multi-level (video, scene, object) language captions via contrastive loss, with captions added to inter-frame embeddings at test time.
Key results
- On NOCS-REAL275 category-level pose estimation, L4D-Track reaches IoU75 76.0 and 5°2cm 47.7 versus HS-Pose's 74.7 and 46.5, and 10°5cm 85.5 versus 82.7.
- On 3D shape reconstruction (NOCS-REAL275, CD x10^-2), the method averages 0.08 versus 0.13 for the best baseline ShAPO.
- Zero-shot instance-level tracking on YCB-Video reaches average ADD-S 86.1 and runs at roughly 20 FPS on NOCS-REAL275.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use L4D-Track instead of HS-Pose | NOCS-REAL275 category-level pose estimation · NOCS-REAL275 | IoU75: 74.7 -> 76.0 | Helped |
| Use L4D-Track instead of HS-Pose | NOCS-REAL275 category-level pose estimation · NOCS-REAL275 | 5°2cm: 46.5 -> 47.7 | Helped |
| Use L4D-Track instead of HS-Pose | NOCS-REAL275 category-level pose estimation · NOCS-REAL275 | 10°5cm: 82.7 -> 85.5 | Helped |
| Use L4D-Track instead of se(3)-TrackNet | YCB-Video pose tracking · YCB-Video | ADD: 87.8 -> 80.4 | Hurt |
| Use L4D-Track instead of se(3)-TrackNet | YCB-Video pose tracking · YCB-Video | ADD-S: 95.5 -> 86.1 | Hurt |
| Use L4D-Track instead of ShAPO | NOCS-REAL275 shape reconstruction · NOCS-REAL275 | CD (10−2): 0.13 -> 0.08 | Helped |
| Remove the Neural Pose-aligned Fields (NPFs) module (ablation) | L4D-Track · NOCS-REAL275 | CD: 0.08 -> 0.36 | Hurt |
| Remove Energy-based Pose Hypothesis (EPH) and use a simple MLP for pose recovery (ablation) | L4D-Track · NOCS-REAL275 | IoU25 | Hurt |
| Remove each multi-level caption contrastive loss in turn (ablation) | L4D-Track · NOCS-REAL275 | IoU25: 86.6 -> 86.2; 86.6 -> 85.7; 86.6 -> 86.0 | Hurt |
| Remove each multi-level caption contrastive loss in turn (ablation) | L4D-Track · YCB-Video | ADD: 80.4 -> 72.6; 80.4 -> 74.3; 80.4 -> 75.0 | Hurt |
Surprising findings
- Removing the energy-based pose hypothesis module (w/o EPH) does not drop CD but reduces 10°5cm and IoU25 relative to the full model, and removing the 2D/segmentation branch barely changes results.
- The ablation shows that removing the NPFs module raises CD from 0.08 to 0.36, indicating the field is critical for shape quality.
Limitations
- Trained on only six NOCS categories, so generalization beyond them is weaker than se(3)-TrackNet on YCB-Video.
- Relies on GPT-generated captions from a semi-automatic pipeline whose size and quality are not reported.
- Tracking evaluation assumes ground-truth pose and size in the initial frame.
Experimental setup
Datasets: NOCS-REAL275 YCB-Video
Benchmarks: NOCS-REAL275 YCB-Video
Metrics: IoU25 IoU50 IoU75 5deg2cm 5deg5cm 10deg2cm 10deg5cm Rerr Terr ADD
Baselines: NOCS CASS SDP SGPA FS-Net GPV-Pose RBP-Pose TTA-COPE HS-Pose 6-PACK CAPTRA Mask Fusion
Builds on: NeRF NOCS CAPTRA CLIP LLaMA
Closest papers by meaning
- X4D-SceneFormer: Enhanced Scene Understanding on 4D Point Cloud Videos through Cross-Modal Knowledge Transfer (AAAI 2024 · similarity 0.88)
- More Text, Less Point: Towards 3D Data-Efficient Point-Language Understanding (AAAI 2025 · similarity 0.87)
- 4DP-QA: Scalable QA for 4D Perception in Vision Language Models (CVPR 2026 · similarity 0.87)
- Motion4D: Learning 3D-Consistent Motion and Semantics for 4D Scene Understanding (NeurIPS 2025 · similarity 0.86)
- Fast Encoder-Based 3D from Casual Videos via Point Track Processing (NeurIPS 2024 · similarity 0.86)
- OpenDlign: Open-World Point Cloud Understanding with Depth-Aligned Images (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.