Research map / Video / Video generation
GEM: A Generalizable Ego-Vision Multimodal World Model for Fine-Grained Ego-Motion, Object Dynamics, and Scene Composition Control
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Presents GEM, a multimodal ego-vision world model built on Stable Video Diffusion that generates RGB and depth video conditioned on ego-trajectories, DINOv2 object tokens, and human poses, with autoregressive noise scheduling for long videos.
Problem
Existing ego-vision world models mainly offer ego-centric control and single-domain, single-modality generation, limiting fine-grained control over object dynamics, human poses, and long-horizon consistency.
Approach
Fine-tunes SVD with three control pathways (ego-trajectory via cross-attention LoRA, sparse DINOv2 tokens with identity embeddings via ObjectNet, human skeletons via PoseNet), jointly generates depth alongside RGB using pseudo-labels, and uses a dynamic per-frame autoregressive noise schedule for long generations.
Key results
- On nuScenes, GEM reaches FVD 158.5 vs Vista 167.7 (reproduced), and on OpenDV FVD 131 vs 163, while FID is slightly worse on nuScenes (10.5 vs 6.6).
- Object motion control reduces COM error from 38.8 to 12.2 on nuScenes and from 55.2 to 11.5 on OpenDV, about 68.8% and 79% improvement.
- Human evaluation on 15s videos strongly prefers GEM over Vista (e.g., 81 vs 27 for temporal consistency).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use GEM instead of Vista | ego-vision video generation · NuScenes | FID: 6.6? -> 10.5 | Hurt |
| use GEM instead of Vista | ego-vision video generation · NuScenes | FVD: 167.7? -> 158.5 | Helped |
| use GEM instead of Vista | ego-vision video generation · OpenDV | FID: 5.5? -> 6.3 | Hurt |
| use GEM instead of Vista | ego-vision video generation · OpenDV | FVD: 163? -> 131 | Helped |
| add control signals (ablation) | GEM · NuScenes | ADE: 3.24 -> 3.07 | Helped |
| add control signals (ablation) | GEM · Nuscenessub | ADE: 3.59 -> 2.85 | Helped |
| add control signals (ablation) | GEM · OpenDV | ADE: 5.39 -> 3.47 | Helped |
| add control signals (ablation) | GEM · NuScenes | COM: 38.8 -> 12.2 | Helped |
| add control signals (ablation) | GEM · OpenDV | COM: 55.2 -> 11.5 | Helped |
| add control signals (ablation) | GEM · OpenDV | AP@IoU=.5, area=all: 0.00 -> 0.12 | Helped |
| add control signals (ablation) | GEM · OpenDV | AP@IoU=.5:.95, area=large: 0.00 -> 0.12 | Helped |
Surprising findings
- For short 2.5s videos, human raters found little difference between GEM and Vista, with most noting no distinction.
Limitations
- Long-horizon generation quality and consistency still need improvement, as acknowledged by the authors.
- FID on nuScenes is worse than Vista, partly attributed to Vista's fine-tuning on nuScenes.
- Evaluation relies on pseudo-labels (depth, ego-trajectories, poses) for ground truth.
Experimental setup
Datasets: OpenDV BDD nuScenes Driving Dojo Honda HDD Honda HAD Drive360 D2City DoTAS CarCrashDataset EgoExo4D self-collected drone YouTube videos
Benchmarks: nuScenes validation OpenDV validation
Metrics: FID FVD ADE COM AP AbsRel human preference
Baselines: Vista DriveGAN DriveDreamer DriveDreamer-2 WoVoGen Drive-WM GenAD
Builds on: Stable Video Diffusion Vista EDM ControlNet-style conditioning
Closest papers by meaning
- EgoGen: An Egocentric Synthetic Data Generator (CVPR 2024 · similarity 0.87)
- EgoVid-5M: A Large-Scale Video-Action Dataset for Egocentric Videos Generation (NeurIPS 2025 · similarity 0.86)
- Learning to Evolve Scenes: Reasoning about Human Activities with Scene Graphs (NeurIPS 2026 · similarity 0.86)
- Generating Multimodal Driving Scenes via Next-Scene Prediction (CVPR 2025 · similarity 0.86)
- FRAME: Floor-aligned Representation for Avatar Motion from Egocentric Video (CVPR 2025 · similarity 0.85)
- EgoBench: An Interactive Egocentric Multimodal Benchmark for Tool-Using Agents (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.