Research map / Video / Video generation
Robust Dreamer: Deviation-Aware Latent Gaussian Memory for Action-Controlled AR Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Robust Dreamer, an action-controlled autoregressive video generator with a Latent Gaussian Memory (latent features stored on Gaussian primitives and recalled by latent splatting) and Deviation Learning with a Dynamic Deviation Archive that injects one-step-synthesized rollout deviations into training memory.
Problem
Long-horizon frame-wise action-controlled image-to-video generation drifts because 3D memories repeatedly cross the latent-RGB VAE interface and training uses clean memory while inference uses corrupted, self-generated memory.
Approach
Store and recall diffusion latents directly on Gaussian primitives via latent-space splatting to avoid repeated VAE decode-encode, and train the generator on realistic corrupted memory by sampling one-step-approximated latent deviations from a stage- and timestep-indexed archive.
Key results
- On ScanNet 300-frame long-term generation, Robust Dreamer reports PSNR 13.43, SSIM 0.422, LPIPS 0.381, FID 38.42, the best among compared methods (best baseline WWarp PSNR 11.76).
- On OmniWorldGame, reports PSNR 17.43 and FID 21.92, versus 16.12 and 36.85 for the strongest baseline (WWarp).
- Ablation (Row A) shows removing latent memory (RGB cycling) causes a significant drop; Row C shows Gaussian-noise deviations underperform the one-step synthesized deviations.
Surprising findings
- The one-step deviation approximation captures only about 58.7% of the error magnitude obtained by explicit ODE simulation.
- Simple blur or color-shift corruptions perform substantially worse than the archive-based deviations.
Limitations
- Performance depends on the quality of the underlying 3D reconstruction; fails under severe occlusion, reflections, or rapid motion.
- Framework is not end-to-end; memory construction and generation are separate modules.
- Non-rigid and highly dynamic scenes (fluids, fire, topology changes) are not well handled; large corruptions may fall outside the archive distribution.
Experimental setup
Datasets: ScanNet DL3DV OmniWorldGame
Benchmarks: ScanNet DL3DV OmniWorldGame
Metrics: PSNR SSIM LPIPS FID
Baselines: MotionCtrl CameraCtrl Self Forcing WorldMem WorldWarp VMem
Builds on: Wan2.1-I2V-14B CUT3R 3D Gaussian Splatting Wan-Move Self Forcing Scaffold-GS
Closest papers by meaning
- Memory as Dynamics: Learning Reliability-Guided Predictive Models for Online Video Perception (ICML 2026 · similarity 0.87)
- Physics-Informed Video Generation via Mixture-of-Experts Latent Alignment (NeurIPS 2026 · similarity 0.86)
- HiCoM: Hierarchical Coherent Motion for Dynamic Streamable Scenes with 3D Gaussian Splatting (NeurIPS 2024 · similarity 0.86)
- AR-Diffusion: Asynchronous Video Generation with Auto-Regressive Diffusion (CVPR 2025 · similarity 0.86)
- Robust Multi-Object 4D Generation for In-the-wild Videos (CVPR 2025 · similarity 0.86)
- UMAMI: Unifying Masked Autoregressive Models and Deterministic Rendering for View Synthesis (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.