Research map / Vision-language and multimodal / Robot action and navigation
Generating 6DoF Object Manipulation Trajectories from Action Description in Egocentric Vision
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Extracts 28,497 6DoF object manipulation trajectories from ordinary Ego-Exo4D videos with a GPT-4o/SAM/SpaTracker pipeline, then trains VLM and point-cloud LLM models to generate trajectories from action descriptions, evaluated on HOT3D.
Problem
Generating 6DoF object manipulation trajectories from textual action descriptions in egocentric video is limited by the scarcity of precise 3D demonstration data, since motion capture datasets cover few objects and actions.
Approach
Automatically extract 6DoF object trajectories from single-camera egocentric videos (no camera calibration needed) using open-vocabulary segmentation, 3D point tracking, depth-based point cloud registration and SVD rotation, then cast trajectory generation as next-token prediction over discretized pose tokens added to VLM/3D-LLM vocabularies.
Key results
- PointLLM (7B) achieves 3D position ADE 0.271 and FDE 0.458 under 6DoF training on HOT3D, outperforming the Seq2Seq baseline by about 0.10 m ADE.
- Sampling 10 trajectories from PointLLM lowers 3D ADE to 0.212, showing multiple plausible trajectories can be generated.
- Adding trajectory information raises verb similarity for BLIP-2 captioning from 58.59 to 67.58 and for PointLLM from 43.05 to 53.33.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use PointLLM with point-cloud input instead of image-based Seq2Seq | 6DoF trajectory generation · HOT3D test | 3D position ADE: 0.374 -> 0.271 | Helped |
| add depth input to BLIP-2 (2.7B) (ablation) | BLIP-2 (2.7B) · HOT3D test, 6DoF | 3D position ADE: 0.280 -> 0.275 | Helped |
| add depth input to BLIP-2 (2.7B) (ablation) | BLIP-2 (2.7B) · HOT3D test, 6DoF | rotation GD: 0.545 -> 0.543 | No clear effect |
| train USST on EgoTraj from scratch (ablation) | USST · HOT3D test | 3D position ADE: 0.303 vs 0.319 | No clear effect |
| fine-tune pretrained USST on EgoTraj (ablation) | USST pretrained on H2O and EgoPAT3D · HOT3D test | 3D position ADE: 0.303 -> 0.254 | Helped |
| increase PointLLM trajectory samples from 1 to 10 (ablation) | PointLLM · HOT3D test | 3D position ADE: 0.271 -> 0.212 | Helped |
| increase PointLLM trajectory samples from 1 to 10 (ablation) | PointLLM · HOT3D test | rotation GD: 0.541 -> 0.540 | No clear effect |
| increase training data scale (ablation) | PointLLM · HOT3D test | 3D position ADE | Helped |
| use trajectory information for caption generation (ablation) | BLIP-2 · HOT3D image and 3D captioning | verb similarity (Simv): 58.59 -> 67.58 | Helped |
| use trajectory information for caption generation (ablation) | PointLLM · HOT3D image and 3D captioning | verb similarity (Simv): 43.05 -> 53.33 | Helped |
Surprising findings
- Larger LLMs do not consistently improve trajectory prediction; BLIP-2 (2.7B) outperforms VILA (8B).
- Rotation prediction (geodesic distance) improves little with more samples, indicating rotation is harder to generate accurately than position.
Limitations
- Cannot handle deformable objects such as clothes since objects are modeled as 6DoF bounding boxes.
- Extraction fails when similar objects are present (segmentation) or when camera pose changes abruptly (registration).
Experimental setup
Datasets: Ego-Exo4D HOT3D H2O EgoPAT3D
Benchmarks: HOT3D
Metrics: ADE FDE Geodesic Distance METEOR BLEU-4 Simv
Baselines: Seq2Seq USST BLIP-2 VILA MiniGPT-3D PointLLM
Builds on: BLIP-2 VILA PointLLM MiniGPT-3D USST SpaTracker
Closest papers by meaning
- EgoVid-5M: A Large-Scale Video-Action Dataset for Egocentric Videos Generation (NeurIPS 2025 · similarity 0.88)
- Robots Pre-train Robots: Manipulation-Centric Robotic Representation from Large-Scale Robot Datasets (ICLR 2025 · similarity 0.87)
- OASIS: Observation-Action Space Alignment via SE(3) Trajectory Prediction for Robotic Manipulation (NeurIPS 2026 · similarity 0.87)
- OmniManip: Towards General Robotic Manipulation via Object-Centric Interaction Primitives as Spatial Constraints (CVPR 2025 · similarity 0.87)
- AffordDP: Generalizable Diffusion Policy with Transferable Affordance (CVPR 2025 · similarity 0.86)
- 6D Object Pose Tracking in Internet Videos for Robotic Manipulation (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.