Research map / Video / Video generation
Video World Models with Long-term Spatial Memory
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Augments an autoregressive video world model (CogVideoX-5B-I2V) with a geometry-grounded long-term spatial memory (TSDF-fused point cloud from CUT3R) plus sparse episodic keyframes and short-term working frames, trained on a 90K-sample MiraData-derived dataset.
Problem
Video world models autoregressively generating frames from actions/camera controls have limited temporal context, so they forget previously generated environments and lose scene consistency on revisits.
Approach
Combines three memory types: short-term working memory (recent latent frames), a long-term spatial memory built by autoregressively predicting point maps and fusing only static parts via TSDF fusion, and a sparse episodic memory of historical reference frames injected via historical cross-attention; static point-cloud renderings condition generation through a ControlNet-style condition DiT.
Key results
- On view recall over reversed trajectories, the method reaches PSNR 19.10, SSIM 0.6471, LPIPS 0.3069, versus 11.71-12.16 PSNR for baselines.
- On VBench, the full model scores best on aesthetic quality (0.5835), temporal flickering (0.7580), motion smoothness (0.9886), and subject consistency (0.9359), but Wan2.1-Inpainting leads imaging quality and background consistency.
- In a 20-person user study, the method ranks highest on camera accuracy (3.6260), static consistency (3.3846), and dynamic plausibility (3.4011).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add geometry-grounded spatial memory and episodic and working memory | CogVideoX-5B-I2V world model · View Recall Consistency | PSNR: 12.01 -> 19.10 | Helped |
| Add geometry-grounded spatial memory and episodic and working memory | CogVideoX-5B-I2V world model · View Recall Consistency | SSIM: 0.4512 -> 0.6471 | Helped |
| Add geometry-grounded spatial memory and episodic and working memory | CogVideoX-5B-I2V world model · View Recall Consistency | LPIPS: 0.5874 -> 0.3069 | Helped |
| Use the proposed model rather than Wan2.1-Inpainting | Video generation on VBench · VBench | Imaging Quality: 0.6788 -> 0.6701 | Hurt |
| Use the proposed model rather than Wan2.1-Inpainting | Video generation on VBench · VBench | Background Consistency: 0.9513 -> 0.9506 | Hurt |
| Add sparse historical keyframes (ablation) | Full model · VBench | Temporal Flickering: 0.7260 -> 0.7580 | Helped |
| Add sparse historical keyframes (ablation) | Full model · VBench | Aesthetic Quality: 0.5603 -> 0.5835 | Helped |
| Add recent context frames (ablation) | Full model · VBench | Temporal Flickering: 0.6740 -> 0.7580 | Helped |
| Add recent context frames (ablation) | Full model · VBench | Motion Smoothness: 0.9862 -> 0.9886 | Helped |
Surprising findings
- Wan2.1-Inpainting achieves higher imaging quality and background consistency than the proposed method, which the authors attribute to it generating relatively static scenes.
Limitations
- TSDF fusion introduces artifacts when revisiting content from camera poses very different from earlier observations.
- Failure with large inter-pose distances or abrupt trajectory angles, where 4D reconstruction fails and spatial memory becomes sparse.
- Drift from error accumulation over time is not addressed.
Experimental setup
Datasets: MiraData
Benchmarks: VBench MiraData test split (500 sequences)
Metrics: PSNR SSIM LPIPS VBench aesthetic quality imaging quality temporal flickering motion smoothness subject consistency background consistency user study AHR (Cam-Acc, Stat-Cons, Dyn-Plaus)
Baselines: TrajectoryCrafter DaS Wan2.1-Inpainting
Builds on: CogVideoX DaS ControlNet CUT3R Mega-SaM
Closest papers by meaning
- WorldMem: Long-term Consistent World Simulation with Memory (NeurIPS 2025 · similarity 0.91)
- Learning 3D Persistent Embodied World Models (NeurIPS 2025 · similarity 0.90)
- Addressable Memory for Video World Models (NeurIPS 2026 · similarity 0.90)
- Learning World Models for Interactive Video Generation (NeurIPS 2025 · similarity 0.89)
- WorldWeaver: Generating Long-Horizon Video Worlds via Rich Perception (NeurIPS 2025 · similarity 0.89)
- Trimming the Long-Tail of Visual World Modeling Evaluation (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.