Research map / Video / Video generation
WorldWeaver: Generating Long-Horizon Video Worlds via Rich Perception
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Jointly models RGB, depth, and optical flow latents in video diffusion/flow models, with a depth-preserving memory bank and group-wise noise scheduling, to reduce drift in long-horizon video generation.
Problem
Long-horizon video generation accumulates structural and motion errors (object warping, drift) when models are trained and conditioned only on RGB signals.
Approach
Jointly predict RGB together with depth and optical flow latents in one unified representation; keep depth noise-free in a long-term memory bank while adding low-level noise to RGB and flow; and assign aligned, segment-wise noise levels to prediction groups during training and streaming denoising at inference.
Key results
- On CogVideoX-2B robotic data, WorldWeaver reaches subject consistency 90.92 and background consistency 92.39, the best among compared long-horizon methods, with drift 0.07 and 30K training steps.
- Versus open-source long-video models, WorldWeaver surpasses Magi 4.5B on drift (0.07 vs 0.09) and background consistency (90.49 vs 90.32), while Magi is higher on subject consistency and image quality.
- Joint depth conditioning raises subject consistency from 92.72 to 94.87 and motion smoothness from 0.74 to 0.82 on 5-second videos.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add perceptual conditions to RGB modeling (ablation) | WorldWeaver on robotic manipulation · Robotic manipulation | Subject consistency: 89.57 -> 90.92 | Helped |
| add perceptual conditions to RGB modeling (ablation) | WorldWeaver on robotic manipulation · Robotic manipulation | Image quality drift: 0.15 -> 0.07 | Helped |
| add depth as a perceptual condition (ablation) | RGB video generation · 5-second videos | Subject consistency: 92.72 -> 94.87 | Helped |
| add depth as a perceptual condition (ablation) | RGB video generation · 5-second videos | Motion smoothness: 0.74 -> 0.82 | Helped |
| add optical flow as a perceptual condition (ablation) | RGB video generation · 5-second videos | Dynamic degree: 0.62 -> 0.74 | Helped |
| add optical flow as a perceptual condition (ablation) | RGB video generation · 5-second videos | Subject consistency: 92.72 -> 92.48 | Hurt |
| add segmentation as a perceptual condition (ablation) | RGB video generation · 5-second videos | Motion smoothness: 0.74 -> 0.70 | Hurt |
| combine depth and optical flow (ablation) | RGB video generation · 5-second videos | Motion smoothness: 0.74 -> 0.85 | Helped |
| combine depth and segmentation (ablation) | RGB video generation · 5-second videos | Subject consistency: 94.87 -> 94.93 | No clear effect |
| set memory noise t_m to 0.9 rather than 0.3 (ablation) | WorldWeaver without perceptual conditions · Robotic manipulation | Image quality drift: 0.10 -> 0.19 | Hurt |
| set memory noise t_m to 0.9 rather than 0.3 (ablation) | WorldWeaver with perceptual conditions · Robotic manipulation | Image quality drift: 0.04 -> 0.07 | Mixed |
Surprising findings
- Optical flow alone mainly improves dynamic degree, while depth gives the larger gains in consistency and motion.
- Rolling Diffusion achieves the lowest drift (0.05) in the comparison table, beating WorldWeaver's 0.07.
Limitations
- Small objects can suddenly disappear in manipulation scenes.
- Success rate decreases and error accumulates as video length grows beyond 10-20 seconds.
- Depth and flow are synthetic from predictive models rather than ground truth.
Experimental setup
Datasets: internal general-purpose video dataset DROID
Benchmarks: VBench
Metrics: subject consistency background consistency image quality motion smoothness dynamic degree start-end quality drift (Delta quality drift) training steps
Baselines: StreamingT2V CausVid SkyReels-V2 Magi History Diffusion Rolling Diffusion Diffusion Forcing FramePack
Builds on: flow matching Diffusion Forcing VideoJAM Video Depth Anything SEA-RAFT
Closest papers by meaning
- Video World Models with Long-term Spatial Memory (NeurIPS 2025 · similarity 0.89)
- Learning World Models for Interactive Video Generation (NeurIPS 2025 · similarity 0.88)
- FreeLong: Training-Free Long Video Generation with SpectralBlend Temporal Attention (NeurIPS 2024 · similarity 0.88)
- Video-Bench: Human-Aligned Video Generation Benchmark (CVPR 2025 · similarity 0.87)
- DepthCrafter: Generating Consistent Long Depth Sequences for Open-world Videos (CVPR 2025 · similarity 0.87)
- Generative Inbetweening through Frame-wise Conditions-Driven Video Generation (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.