Research map / Video / Video generation
EgoVid-5M: A Large-Scale Video-Action Dataset for Egocentric Videos Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces EgoVid-5M, a 5M-clip egocentric video dataset with VIO-based kinematic and LLM-generated text annotations plus a multi-metric cleaning pipeline, and EgoDreamer, an action-conditioned egocentric video generator.
Problem
Egocentric video generation lacks large-scale, high-quality training data with action annotations; existing datasets (Ego4D, Ego-Exo4D) contain excessive camera noise and noisy narrations unsuitable for generative training.
Approach
Builds a generation-oriented egocentric dataset from Ego4D by fusing ParticleSfM camera poses with filtered IMU signals via a Kalman filter for kinematic control, captioning with MLLM+LLM, and cleaning by CLIP, EgoVideo, optical flow, and DOVER thresholds; EgoDreamer adds a Unified Action Encoder and Adaptive Alignment module for joint text and Plücker-embedding control.
Key results
- Finetuning SVD, DynamiCrafter, and OpenSora on EgoVid improves CD-FVD for all three (e.g., OpenSora 809.46 to 718.32; DynamiCrafter 243.63 to 236.82).
- Motion strength rises substantially with EgoVid, e.g., SVD 18.897 to 21.032 and DynamiCrafter 9.357 to 18.329.
- EgoDreamer with UAE and AA reaches CD-FVD 229.82 and rotation/translation errors 3.28/7.62, versus 5.32/9.27 for the ControlNet baseline row.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Finetune SVD on EgoVid | SVD · Egocentric video generation | CD-FVD: 591.61 -> 548.32 | Helped |
| Finetune DynamiCrafter on EgoVid | DynamiCrafter · Egocentric video generation | CD-FVD: 243.63 -> 236.82 | Helped |
| Finetune OpenSora on EgoVid | OpenSora · Egocentric video generation | CD-FVD: 809.46 -> 718.32 | Helped |
| Finetune SVD on EgoVid | SVD · Egocentric video generation | Semantic Consistency: 0.258 -> 0.266 | Helped |
| Finetune DynamiCrafter on EgoVid | DynamiCrafter · Egocentric video generation | Action Consistency: 0.481 -> 0.494 | Helped |
| Finetune OpenSora on EgoVid | OpenSora · Egocentric video generation | Action Consistency: 0.489 -> 0.494 | Helped |
| Add EgoVid training (ablation) | EgoDreamer with ControlNet · EgoDreamer kinematic and action control | CD-FVD: 241.90 -> 238.87 | Helped |
| Add UAE to EgoVid-trained ControlNet (ablation) | EgoDreamer · EgoDreamer kinematic and action control | Action Consistency: 0.493 -> 0.494 | Helped |
| Replace ControlNet with ControlNeXt while using UAE (ablation) | EgoDreamer · EgoDreamer kinematic and action control | CD-FVD: 239.01 -> 234.13 | Helped |
| Replace ControlNet with Adaptive Alignment while using UAE (ablation) | EgoDreamer · EgoDreamer kinematic and action control | CD-FVD: 241.90 -> 229.82 | Helped |
| Use Strategy-1 cleaning thresholds (ablation) | DynamiCrafter · EgoVid-1M-1 | Motion strength | Hurt |
| Use Strategy-2 relaxed cleaning thresholds (ablation) | DynamiCrafter · EgoVid-1M-2 | Motion strength | Helped |
Surprising findings
- Relaxing CLIP and optical-flow thresholds in one cleaning strategy led to excessive motion and visual fragmentation that hurt text-frame consistency.
Limitations
- Kinematic annotations exist only for a 65K subset because of known Ego4D IMU issues.
- Evaluation relies on automatic metrics and fixed pretrained judges (CLIP, EgoVideo, DOVER) without human evaluation.
Experimental setup
Datasets: Ego4D Ego-Exo4D EgoVid-5M
Benchmarks: EgoVid-val (1.2K samples)
Metrics: CD-FVD CLIP semantic consistency EgoVideo action consistency DOVER clarity motion smoothness via AMT interpolation RAFT motion strength rotation error translation error
Baselines: SVD DynamiCrafter OpenSora ControlNet ControlNeXt
Builds on: Ego4D DynamiCrafter Stable Video Diffusion OpenSora ControlNet
Closest papers by meaning
- EgoGen: An Egocentric Synthetic Data Generator (CVPR 2024 · similarity 0.91)
- PlayerOne: Egocentric World Simulator (NeurIPS 2025 · similarity 0.90)
- EgoSim: An Egocentric Multi-view Simulator and Real Dataset for Body-worn Cameras during Motion and Activity (NeurIPS 2024 · similarity 0.90)
- EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real World (CVPR 2024 · similarity 0.90)
- Exocentric-to-Egocentric Video Generation (NeurIPS 2024 · similarity 0.90)
- Action Scene Graphs for Long-Form Understanding of Egocentric Videos (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.