Research map / Video / Video generation
PlayerOne: Egocentric World Simulator
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PlayerOne, an egocentric world simulator that generates egocentric video from one first-frame image and exocentric-captured human motion, using part-disentangled motion injection and joint 4D point-map reconstruction, trained coarse-to-fine and distilled for 8 FPS.
Problem
Existing world simulators are mostly game-like, limit users to directional actions, or lack human motion control, so users cannot freely act in realistic egocentric scenes with consistent worlds.
Approach
Drives an egocentric video diffusion transformer with real-scene human motion (SMPL parameters split into body/feet, hands, and head-as-rotation camera) and jointly denoises video latents with 4D scene point-map latents (CUT3R-rendered) during training only, plus coarse-to-fine training on egocentric text-video data and curated ego-exo motion-video pairs.
Key results
- On the self-built benchmark, PlayerOne reaches FVD 226.12 and MPJPE 127.16, beating Cosmos (Diff-14B) at FVD 302.17 and MPJPE 256.73.
- Pretraining on egocentric text-video data improves CLIP-Score from 65.6 (baseline) to 74.4, and PMI raises it further to 81.3.
- In a user study with 20 annotators, PlayerOne scores 3.86 on Alignment versus 2.71 for Cosmos (Diff-14B).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add egocentric text-video pretraining (ablation) | Wanx2.1 1.3B baseline · Nymeria benchmark | DINO-Score: 51.3 -> 56.6 | Helped |
| add part-disentangled motion injection (ablation) | pretrained PlayerOne · Nymeria benchmark | DINO-Score: 58.0 -> 62.5 | Helped |
| add the camera encoder (ablation) | pretrained PlayerOne with PMI · Nymeria benchmark | DINO-Score: 60.7 -> 62.5 | Helped |
| use part-disentangled motion injection instead of ControlNet injection (ablation) | pretrained PlayerOne · Nymeria benchmark | DINO-Score: 57.1 -> 62.5 | Helped |
| add motion-video data filtering (ablation) | pretrained PlayerOne with PMI · Nymeria benchmark | DINO-Score: 62.5 -> 64.2 | Helped |
| add scene-frame reconstruction with adapter and CUT3R (ablation) | pretrained PlayerOne with PMI and filtering · Nymeria benchmark | DINO-Score: 64.2 -> 67.8 | Helped |
| add the reconstruction adapter (ablation) | pretrained PlayerOne with PMI, filtering, and reconstruction · Nymeria benchmark | DINO-Score: 62.7 -> 67.8 | Helped |
| replace CUT3R with DUSt3R for point-map rendering (ablation) | pretrained PlayerOne with PMI and filtering · Nymeria benchmark | DINO-Score: 67.5 -> 67.8 | No clear effect |
| use coarse-to-fine training rather than training only on motion-video pairs or joint one-stage training (ablation) | PlayerOne · Nymeria benchmark | motion alignment | Helped |
| omit scene-frame reconstruction (ablation) | PlayerOne · Nymeria benchmark | scene consistency | Hurt |
| compare PlayerOne with Cosmos (Diff-14B) | Nymeria benchmark · Nymeria benchmark | DINO-Score: 51.6 -> 67.8 | Helped |
Limitations
- Performance on game scenarios is slightly inferior to realistic scenarios, attributed to imbalanced training data.
- No public benchmark exists; the authors built a 100-video benchmark from Nymeria themselves, and no method shares the same setting so comparisons are limited to Cosmos and Aether.
- Training data is derived from limited ego-exo datasets, and the motion-video pairs are small in scale.
Experimental setup
Datasets: Egovid-5M EgoExo-4D Nymeria FT-HID EgoExo-Fitness
Benchmarks: PlayerOne benchmark (100 Nymeria videos)
Metrics: DINO-Score CLIP-Score MPJPE MRRPE FVD LPIPS SSIM frame consistency user study ratings
Baselines: Aether Cosmos (Diff-7B) Cosmos (Diff-14B) ControlNet-based motion injection DUSt3R-based point map rendering
Builds on: Wan2.1 CUT3R SMPLest-X Plücker ray camera embedding ControlNet
Closest papers by meaning
- EgoVid-5M: A Large-Scale Video-Action Dataset for Egocentric Videos Generation (NeurIPS 2025 · similarity 0.90)
- EgoGen: An Egocentric Synthetic Data Generator (CVPR 2024 · similarity 0.89)
- The Matrix: Infinite-Horizon World Generation with Real-Time Moving Control (NeurIPS 2025 · similarity 0.86)
- Grounded-Exo2Ego: Structured Semantic Grounding for Robust Exocentric-to-Egocentric Video Generation (NeurIPS 2026 · similarity 0.86)
- Exocentric-to-Egocentric Video Generation (NeurIPS 2024 · similarity 0.86)
- EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real World (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.