Research map / Video / Video generation
Exocentric-to-Egocentric Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Exo2Ego-V generates egocentric videos from four 360°-configured exocentric videos using a diffusion pipeline with a multi-view exocentric encoder, a PixelNeRF-based view translation prior, and temporal layers.
Problem
Generating egocentric videos of daily-life skilled human activities from sparse, widely spread 4-view exocentric videos, which is hard due to large viewpoint differences, complex dynamic motion, and real-world environments.
Approach
A trainable copy of the ego UNet encodes multi-view exo videos as appearance conditions via concatenated self-attention, and a generalizable PixelNeRF renders spatially aligned ego features concatenated to the noisy ego latents as a geometric prior, with temporal layers added for frame consistency.
Key results
- On Ego-Exo4D unseen actions, Exo2Ego-V reaches LPIPS 0.249 on basketball and 0.235 on covid test versus 0.397 and 0.477 for SVD, with the abstract claiming an average 35% LPIPS improvement.
- On H2O, Exo2Ego-V achieves PSNR 18.600, SSIM 0.581, LPIPS 0.189 versus SVD's 16.530, 0.468, 0.271.
- Ablation on cooking (unseen action): full model PSNR 17.367 and LPIPS 0.408; removing the exo2ego prior lowers LPIPS to 0.364 but reduces PSNR to 17.268 and misses limbs in qualitative results.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Exo2Ego-V instead of SVD | Exo2Ego-V · Ego-Exo4D categories, unseen actions | LPIPS: average of 35% | Helped |
| remove multi-view exocentric encoder (ablation) | Exo2Ego-V · Cooking, unseen action | PSNR: 17.367 -> 16.777 | Hurt |
| remove Exo2Ego view translation prior (ablation) | Exo2Ego-V · Cooking, unseen action | PSNR: 17.367 -> 17.268 | Hurt |
| remove Exo2Ego view translation prior (ablation) | Exo2Ego-V · Cooking, unseen action | LPIPS: 0.408 -> 0.364 | Helped |
| add temporal layers to the exocentric encoder (ablation) | Exo2Ego-V · Cooking, unseen action | LPIPS: 0.408 -> 0.507 | Hurt |
| use 3 exocentric views instead of 4 (ablation) | Exo2Ego-V · Cooking, unseen action | LPIPS: 0.408 -> 0.383 | Hurt |
| use 2 exocentric views instead of 4 (ablation) | Exo2Ego-V · Cooking, unseen action | LPIPS: 0.408 -> 0.399 | Hurt |
| use 1 exocentric view instead of 4 (ablation) | Exo2Ego-V · Cooking, unseen action | LPIPS: 0.408 -> 0.395 | Hurt |
| replace the exocentric encoder with CLIP features (ablation) | Exo2Ego-V · Cooking, unseen action | LPIPS: 0.408 -> 0.425 | Hurt |
| use temporal-spatial rather than spatial-temporal attention (ablation) | Exo2Ego-V · Cooking, unseen action | LPIPS: 0.408 -> 0.402 | Mixed |
Surprising findings
- Removing the Exo2Ego view translation prior gives the best LPIPS on cooking but yields inaccurate egocentric content, so LPIPS alone is misleading.
- PixelNeRF gets competitive PSNR despite blurry outputs, since PSNR favors blur.
Limitations
- Focuses on several categories of skilled activities; not yet general activities.
- Inference is not real-time (9.06 s per 8-frame clip).
- Unseen-scene performance is limited; the authors note the difficulty of generalizing to new environments with few scenes.
Experimental setup
Datasets: Ego-Exo4D H2O
Benchmarks: Ego-Exo4D H2O
Metrics: PSNR SSIM LPIPS
Baselines: PixelNeRF Stable Diffusion Stable Video Diffusion Exo2Ego (image-level)
Builds on: Stable Diffusion AnimateDiff PixelNeRF ReconFusion MVDream
Closest papers by meaning
- Grounded-Exo2Ego: Structured Semantic Grounding for Robust Exocentric-to-Egocentric Video Generation (NeurIPS 2026 · similarity 0.91)
- EgoVid-5M: A Large-Scale Video-Action Dataset for Egocentric Videos Generation (NeurIPS 2025 · similarity 0.90)
- EgoExo-Gen: Ego-centric Video Prediction by Watching Exo-centric Videos (ICLR 2025 · similarity 0.87)
- EgoGen: An Egocentric Synthetic Data Generator (CVPR 2024 · similarity 0.87)
- Bootstrap Your Own Views: Masked Ego-Exo Modeling for Fine-grained View-invariant Video Representations (CVPR 2025 · similarity 0.87)
- Viewpoint Rosetta Stone: Unlocking Unpaired Ego-Exo Videos for View-invariant Representation Learning (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.