Research map / Video / Video segmentation and tracking
FIction: 4D Future Interaction Prediction from Video
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces 4D future interaction prediction (where objects will be interacted with in 3D and how the body will pose) from egocentric video plus 3D scene voxels, with the FICTION transformer+CVAE model and a dataset built from Ego-Exo4D.
Problem
Predicting which 3D object locations a person will interact with over the next minutes and the body poses used during those interactions, which existing 2D video or short-horizon 3D pose methods do not model.
Approach
A multimodal transformer encodes past egocentric video features, SMPL pose, and a voxelized 3D object/actor layout; a linear decoder predicts future interaction voxels, and a CVAE conditioned on the query location generates location-specific body pose distributions.
Key results
- FICTION reaches PR-AUC 21.0 on cooking location prediction vs 16.9 for the best baseline (OCT), with Chamfer distance 7.4 vs 9.0.
- On pose prediction in cooking, FICTION achieves MPJPE 229 mm vs 264 mm for fine-tuned T2M-GPT, and PA-MPJPE 56 vs 61.
- The paper reports more than 30% relative gains over the best baseline across tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add video input (ablation) | FICTION · Cooking | PR-AUC: 20.0 -> 21.0 | Helped |
| add video input (ablation) | FICTION · Bike Repair | PR-AUC: 18.7 -> 18.7 | No clear effect |
| add video input (ablation) | FICTION · Health | PR-AUC: 12.0 -> 12.7 | Helped |
| add body-pose input (ablation) | FICTION · Cooking | PR-AUC: 19.1 -> 21.0 | Helped |
| add body-pose input (ablation) | FICTION · Bike Repair | PR-AUC: 18.3 -> 18.7 | Helped |
| add body-pose input (ablation) | FICTION · Health | PR-AUC: 11.7 -> 12.7 | Helped |
| add environment context (ablation) | FICTION · Cooking | PR-AUC: 9.9 -> 21.0 | Helped |
| add environment context (ablation) | FICTION · Bike Repair | PR-AUC: 6.0 -> 18.7 | Helped |
| add environment context (ablation) | FICTION · Health | PR-AUC: 4.7 -> 12.7 | Helped |
| use FICTION instead of OCT | future interaction location prediction · Cooking | PR-AUC: 16.9 -> 21.0 | Helped |
| use FICTION instead of OCT | future interaction location prediction · Bike Repair | Chamfer distance: 9.4 -> 7.2 | Helped |
| use FICTION instead of OCT | future interaction location prediction · Health | PR-AUC: 11.2 -> 12.7 | Helped |
Surprising findings
- Removing the 3D environment context causes the largest drop in location prediction (cooking PR-AUC falls from 21.0 to 9.9), more than removing video or pose.
Limitations
- Health scenario performance is lower because the object vocabulary has few relevant objects.
- Assumes static objects; dynamic object movement is not modeled.
- Relies on off-the-shelf 3D object detection, pose estimation, and LLM-based narration matching for dataset creation.
Experimental setup
Datasets: Ego-Exo4D
Benchmarks: Ego-Exo4D future interaction prediction (cooking, bike repair, health)
Metrics: PR-AUC Chamfer distance MPJPE PA-MPJPE
Baselines: HierVL OCT 4D-Humans T2M-GPT T2M-GPT-FT VoxFormer OccFormer Video-to-pose CVAE
Builds on: VoxFormer OccFormer T2M-GPT CVAE WHAM Detic
Closest papers by meaning
- Summarize the Past to Predict the Future: Natural Language Descriptions of Context Boost Multimodal Object Interaction Anticipation (CVPR 2024 · similarity 0.87)
- InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interaction (NeurIPS 2024 · similarity 0.85)
- FutureHuman3D: Forecasting Complex Long-Term 3D Human Behavior from Video Observations (CVPR 2024 · similarity 0.85)
- Gaze Beyond the Frame: Forecasting Egocentric 3D Visual Span (NeurIPS 2025 · similarity 0.84)
- Multimodal Sense-Informed Forecasting of 3D Human Motions (CVPR 2024 · similarity 0.84)
- PlaySlot: Learning Inverse Latent Dynamics for Controllable Object-Centric Video Prediction and Planning (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.