Research map / Video / Video segmentation and tracking
WorldMem: Long-term Consistent World Simulation with Memory
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes WorldMem, an autoregressive interactive world simulator that stores token-level memory units (frames, poses, timestamps) and retrieves them via FOV/time confidence with state-aware cross-attention to maintain long-term 3D consistency.
Problem
Video world simulators with limited context windows forget previously generated content, causing 3D spatial and temporal inconsistency when the camera returns to earlier scenes or when time passes.
Approach
A token-level memory bank of past latent frames with poses and timestamps, retrieved by FOV-overlap and time-proximity confidence, and injected through state-aware cross-attention (Plücker pose and sinusoidal time embeddings, relative state formulation) on top of a conditional DiT with Diffusion Forcing.
Key results
- On RealEstate10K, WorldMem reaches PSNR 23.34, LPIPS 0.1672, rFID 43.14, beating Viewcrafter (21.72 / 0.1729 / 58.43) and DFoT (16.42 / 0.2933 / 110.34).
- Beyond the context window on Minecraft, the memory-augmented model consistently outperforms Diffusion Forcing across PSNR, LPIPS and rFID after 600-frame memory bank.
- Ablation: progressive memory sampling during training improves PSNR to 23.98 and rFID to 15.37 over small- or large-range sampling.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the memory bank and state-aware memory attention | DFoT on RealEstate10K · RealEstate10K | PSNR: 21.72 -> 23.34 | Helped |
| Add the memory bank and state-aware memory attention | DFoT on RealEstate10K · RealEstate10K | LPIPS: 0.1729 -> 0.1672 | Helped |
| Add the memory bank and state-aware memory attention | DFoT on RealEstate10K · RealEstate10K | rFID: 58.43 -> 43.14 | Helped |
| Use progressive sampling rather than small-range sampling (ablation) | WorldMem on Minecraft · Minecraft | PSNR: 19.23 -> 23.98 | Helped |
| Use progressive sampling rather than small-range sampling (ablation) | WorldMem on Minecraft · Minecraft | LPIPS: 0.3786 -> 0.1429 | Helped |
| Use progressive sampling rather than small-range sampling (ablation) | WorldMem on Minecraft · Minecraft | rFID: 46.55 -> 15.37 | Helped |
| Use progressive sampling rather than large-range sampling (ablation) | WorldMem on Minecraft · Minecraft | PSNR: 21.11 -> 23.98 | Helped |
| Use progressive sampling rather than large-range sampling (ablation) | WorldMem on Minecraft · Minecraft | LPIPS: 0.3855 -> 0.1429 | Helped |
| Use progressive sampling rather than large-range sampling (ablation) | WorldMem on Minecraft · Minecraft | rFID: 42.96 -> 15.37 | Helped |
| Use memory context length 8 instead of 1 (ablation) | WorldMem on Minecraft · Minecraft | PSNR: 22.18 -> 25.32 | Helped |
| Use memory context length 8 instead of 1 (ablation) | WorldMem on Minecraft · Minecraft | LPIPS: 0.1899 -> 0.1429 | Helped |
| Use memory context length 8 instead of 1 (ablation) | WorldMem on Minecraft · Minecraft | rFID: 20.47 -> 15.37 | Helped |
Surprising findings
- Random memory retrieval causes sharp rFID degradation and rapid divergence from ground truth, showing retrieval quality matters more than memory size alone.
- Memory context length 16 underperforms length 8, suggesting excessive memory frames add noise or reduce retrieval precision.
Limitations
- Cannot guarantee retrieval of all necessary information; view overlap alone may fail when views are blocked by obstacles.
- Interactions in the environment lack diversity and realism; RealEstate evaluation uses short non-interactive clips.
- Memory usage grows linearly with sequence length.
Experimental setup
Datasets: MineDojo RealEstate10K
Benchmarks: Minecraft benchmark (MineDojo-based) RealEstate10K
Metrics: PSNR LPIPS rFID
Baselines: Full sequence diffusion transformer Diffusion Forcing CameraCtrl TrajAttn Viewcrafter DFoT
Builds on: Diffusion Forcing Oasis DFoT Conditional Diffusion Transformer
Closest papers by meaning
- Video World Models with Long-term Spatial Memory (NeurIPS 2025 · similarity 0.91)
- Learning 3D Persistent Embodied World Models (NeurIPS 2025 · similarity 0.89)
- StateSpaceDiffuser: Bringing Long Context to Diffusion World Models (NeurIPS 2025 · similarity 0.87)
- From Zero to Hero: Training-Free Custom Concept Spawning in World Models (NeurIPS 2026 · similarity 0.87)
- Trimming the Long-Tail of Visual World Modeling Evaluation (NeurIPS 2026 · similarity 0.87)
- Addressable Memory for Video World Models (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.