Research map / Video / Video segmentation and tracking
EgoExo-Gen: Ego-centric Video Prediction by Watching Exo-centric Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EgoExo-Gen, which predicts ego-centric hand-object masks from exo-centric video and then drives a video diffusion model with those masks to generate future ego-centric frames.
Problem
Cross-view video prediction: generating future ego-centric frames from a synchronized exo-centric video, the first ego frame, and a text instruction, without camera poses or depth.
Approach
A cross-view mask prediction model with ego-exo memory attention anticipates future ego hand-object masks, which then condition an SEINE-based video diffusion model; an automated SAM-2 and foundation-model pipeline creates pseudo HOI masks for training.
Key results
- EgoExo-Gen reaches 0.537 SSIM, 18.395 PSNR, 0.311 LPIPS and 1031.7 FVD on Ego-Exo4D, beating the best baseline ConsistI2V (0.532 SSIM, 18.318 PSNR, 1109.3 FVD).
- On zero-shot transfer to H2O, EgoExo-Gen gets 0.486 SSIM and 1360.5 FVD versus 1657.9 FVD for DynamiCrafter.
- Hand-object mask conditioning improves SEINE from 1063.5 to 836.0 FVD when using ground-truth future masks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add predicted HOI masks as structural guidance | EgoExo-Gen · Ego-Exo4D | SSIM: 0.518 -> 0.537 | Helped |
| add predicted HOI masks as structural guidance | EgoExo-Gen · Ego-Exo4D→H2O | FVD: 1797.965 -> 1360.477 | Helped |
| add hand and object masks rather than no mask condition (ablation) | SEINE video prediction · Ego-Exo4D | SSIM: 0.518 -> 0.571 | Helped |
| use random masks instead of HOI masks (ablation) | SEINE video prediction · Ego-Exo4D | FVD: 1063.458 -> 1089.458 | Hurt |
| add object masks only (ablation) | SEINE video prediction · Ego-Exo4D | SSIM: 0.518 -> 0.548 | Helped |
| add hand masks only (ablation) | SEINE video prediction · Ego-Exo4D | SSIM: 0.518 -> 0.565 | Helped |
| distinguish left and right hands in the masks (ablation) | SEINE video prediction · Ego-Exo4D | SSIM: 0.571 -> 0.569 | Hurt |
| remove text while retaining first-frame and HOI conditions (ablation) | SEINE video prediction · Ego-Exo4D | SSIM: 0.571 -> 0.555 | Hurt |
| remove the first-frame condition (ablation) | SEINE video prediction · Ego-Exo4D | SSIM: 0.571 -> 0.229 | Hurt |
| remove HOI condition (ablation) | SEINE video prediction · Ego-Exo4D | SSIM: 0.571 -> 0.518 | Hurt |
| remove exo-memory attention (ablation) | EgoExo-Gen cross-view mask predictor · Ego-Exo4D | IoU%: 20.315 -> 4.630 | Hurt |
| remove ego-memory attention (ablation) | EgoExo-Gen cross-view mask predictor · Ego-Exo4D | IoU%: 20.315 -> 14.655 | Hurt |
Surprising findings
- Using exo-centric RGB or exo HOI masks directly as conditions gives only marginal gains, while ego-view HOI masks give much larger improvements.
- With predicted (not future-oracle) masks, ego-centric segmentation IoU on invisible frames is only about 20%, indicating mask prediction is the bottleneck.
Limitations
- Predicted masks for future invisible frames are of low quality, limiting generation quality.
- Performance degrades on actions with complex hand movement, where both EgoExo-Gen and ConsistI2V fail.
- Evaluated only on the cooking scenario of Ego-Exo4D and tabletop H2O clips of short duration.
Experimental setup
Datasets: Ego-Exo4D H2O
Benchmarks: Ego-Exo4D H2O
Metrics: SSIM PSNR LPIPS FVD IoU CA LE
Baselines: SVD Seer DynamiCrafter SparseCtrl SEINE ConsistI2V
Builds on: SEINE SAM-2 EgoHOS ConsistI2V CBAM
Closest papers by meaning
- Exocentric-to-Egocentric Video Generation (NeurIPS 2024 · similarity 0.87)
- Bootstrap Your Own Views: Masked Ego-Exo Modeling for Fine-grained View-invariant Video Representations (CVPR 2025 · similarity 0.87)
- EgoVid-5M: A Large-Scale Video-Action Dataset for Egocentric Videos Generation (NeurIPS 2025 · similarity 0.87)
- Retrieval-Augmented Egocentric Video Captioning (CVPR 2024 · similarity 0.86)
- EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real World (CVPR 2024 · similarity 0.86)
- Whole-Body Conditioned Egocentric Video Prediction (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.