Research map / Video / Video generation
Grounded-Exo2Ego: Structured Semantic Grounding for Robust Exocentric-to-Egocentric Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Grounded-Exo2Ego, a dual-branch video diffusion model for single-exo-video to ego-video generation that adds object-level semantic grounding (gated cross-attention with reprojected masks), ego camera re-localization, and a synthetic data engine.
Problem
Generating egocentric video from a single exocentric video is hard because monocular geometric renderings are unreliable under extreme viewpoint change and occlusion, and real training data is misaligned between ego camera poses and monocular reconstructions.
Approach
Combine geometric anchoring (rendered point-cloud conditioning as separate tokens) with semantic grounding that routes per-object text token packets to spatially matching video tokens via reprojected segmentation masks in cross-attention; additionally re-localize ego cameras inside the monocular reconstruction to fix train-time misalignment.
Key results
- On EgoExo4D New Action, Ours reaches PSNR 18.64 dB vs EgoX 16.05 dB, LPIPS 0.346 vs 0.498, and FVD 119.9 vs 184.47.
- On EgoExo4D Unseen, Ours reaches PSNR 16.05 dB vs EgoX 14.38 dB and FVD 385.8 vs 440.64, with object IoU 0.412 vs 0.092.
- Ablations show camera re-localization alone improves all nine metrics over the re-implemented EgoX (LTX, data), e.g. IoU +30.0%/+35.7%.
Surprising findings
- Lifting the backbone and training data alone worsens video metrics (FVD +21.3%/+26.5%, T-LPIPS) relative to EgoX even as PSNR and location error improve.
- Given the exact ego-view attention mask only improves metrics modestly over reprojected masks, while removing object-token gating costs more than any mask perturbation.
Limitations
- Fixed short output (448x448, 33 frames) limited by VRAM; longer videos need chunked generation.
- Performance degrades when the ego view is largely blank in the reconstruction, requiring hallucination; the method depends on off-the-shelf monocular reconstruction.
- Target ego camera trajectory must be estimated at test time, and re-localization is tuned to MoGe2 error characteristics.
Experimental setup
Datasets: EgoExo4D Embody3D Infinigen scenes (synthetic)
Benchmarks: EgoExo4D (Unseen split) EgoExo4D (New Action split)
Metrics: PSNR SSIM LPIPS CLIP-I Location Error IoU Contour Chamfer distance FVD T-LPIPS
Baselines: EgoX Exo2Ego-V TrajectoryCrafter Wan-Fun-Control Wan-VACE Vista4D
Builds on: EgoX IC-LoRA flow matching Vista4D
Closest papers by meaning
- Exocentric-to-Egocentric Video Generation (NeurIPS 2024 · similarity 0.91)
- EgoVid-5M: A Large-Scale Video-Action Dataset for Egocentric Videos Generation (NeurIPS 2025 · similarity 0.89)
- Object-Shot Enhanced Grounding Network for Egocentric Video (CVPR 2025 · similarity 0.88)
- EgoGen: An Egocentric Synthetic Data Generator (CVPR 2024 · similarity 0.88)
- Viewpoint Rosetta Stone: Unlocking Unpaired Ego-Exo Videos for View-invariant Representation Learning (CVPR 2025 · similarity 0.87)
- PlayerOne: Egocentric World Simulator (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.