Research map / Video / Video generation
Sekai: A Video Dataset towards World Exploration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Sekai, a ~5,000-hour worldwide first-person walking and drone video dataset from YouTube and a video game with location, scene, weather, crowd, caption and camera-trajectory annotations, and shows fine-tuning video generators on it improves quality and camera control.
Problem
Existing video generation datasets lack long, diverse, worldwide egocentric footage with exploration annotations (camera trajectories) and world annotations (location, weather, scene), limiting world exploration model training.
Approach
Curate a large egocentric world-exploration video dataset from YouTube walking/drone videos plus ground-truth-annotated game footage, annotate with VLM and SfM pipelines, and add a diversity-aware sampler (quality, location, category, camera trajectory) to produce a balanced high-quality subset (Sekai-Real-HQ).
Key results
- Fine-tuning SkyReels-V2-T2V-14B on Sekai-Real-HQ raised VBench overall score from 4.26 (5k steps) to 4.34 (20k steps); I2V overall rose from 6.10 to 6.29.
- On drone-view Sekai-Real, interactive camera control fine-tuning reduced TransErr from 28.32 to 17.19 and RotErr from 27.2 to 19.89 versus the baseline.
- On Sekai-Game, fine-tuning reduced TransErr from 7.64 to 4.22 and RotErr from 8.36 to 6.22.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune using annotated Sekai-Real camera trajectories | Wan2.1-Fun-V1.1-1.3B-Control-Camera · Sekai-Real drone-view test videos | TransErr: 28.32 -> 17.19 | Helped |
| Fine-tune using annotated Sekai-Real camera trajectories | Wan2.1-Fun-V1.1-1.3B-Control-Camera · Sekai-Real drone-view test videos | RotErr: 27.2 -> 19.89 | Helped |
| Fine-tune using annotated Sekai-Real camera trajectories | Wan2.1-Fun-V1.1-1.3B-Control-Camera · Sekai-Real drone-view test videos | Aesthetic Quality: 58.25% -> 59.18% | Helped |
| Fine-tune using annotated Sekai-Game camera trajectories | Wan2.1-Fun-V1.1-1.3B-Control-Camera · Sekai-Game test videos | TransErr: 7.64 -> 4.22 | Helped |
| Fine-tune using annotated Sekai-Game camera trajectories | Wan2.1-Fun-V1.1-1.3B-Control-Camera · Sekai-Game test videos | RotErr: 8.36 -> 6.22 | Helped |
| Fine-tune on Sekai-Real-HQ for 20000 steps | SkyReels-V2-T2V-14B-540P · Sekai-Real-HQ test set, T2V | Overall Score: 4.26 -> 4.34 | Helped |
| Continue T2V fine-tuning from 10000 to 20000 steps | SkyReels-V2-T2V-14B-540P · Sekai-Real-HQ test set, T2V | Dynamic Degree: 94.67% -> 78.66% | Hurt |
| Fine-tune on Sekai-Real-HQ for 17500 steps | SkyReels-V2-I2V-14B-540P · Sekai-Real-HQ test set, I2V | Overall Score: 6.10 -> 6.29 | Helped |
| Continue I2V fine-tuning from 5000 to 17500 steps | SkyReels-V2-I2V-14B-540P · Sekai-Real-HQ test set, I2V | Dynamic Degree: 100.00% -> 85.00% | Hurt |
Surprising findings
- Dynamic Degree decreased after early training peaks in T2V and I2V, interpreted as a shift from exaggerated motion to sharper, cleaner frames.
- Qwen2.5-VL struggled to detect weather (rain) from frames, with most disagreements between cloudy/foggy and rainy labels.
Limitations
- Weather annotation accuracy is limited for rain, which is hard to see in single frames.
- Location annotations depend on video descriptions; about 8% of clips could not be matched to a chapter and were discarded.
- Full Sekai-Real is too costly to train on, so experiments use only a 400-hour subset.
Experimental setup
Datasets: Sekai-Real Sekai-Game Sekai-Real-HQ OpenVid-1M
Benchmarks: VBench VBench++ CameraCtrl TransErr/RotErr metrics
Metrics: subject consistency background consistency motion smoothness dynamic degree aesthetic quality imaging quality I2V subject I2V background TransErr RotErr
Baselines: OpenVid-1M Wan2.1-Fun-V1.1-1.3B-Control-Camera VGGT DPVO
Builds on: Cosmos MegaSaM VBench CameraCtrl SkyReels-V2
Closest papers by meaning
- WorldModelBench: Judging Video Generation Models As World Models (NeurIPS 2025 · similarity 0.86)
- LoopNav: Benchmarking Spatial Consistency in World Models (NeurIPS 2026 · similarity 0.85)
- WorldReasonBench: Human-Aligned Stress Testing of Video Generators as Future World-State Predictors (NeurIPS 2026 · similarity 0.84)
- WorldVLN: Autoregressive World Action Model for Aerial Vision-Language Navigation (NeurIPS 2026 · similarity 0.84)
- From an Image to a Scene: Learning to Imagine the World from a Million 360° Videos (NeurIPS 2024 · similarity 0.84)
- ChronoMagic-Bench: A Benchmark for Metamorphic Evaluation of Text-to-Time-lapse Video Generation (NeurIPS 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.