AI Paper Atlas

Research map / Probabilistic and scientific modeling / Dynamical systems

Neural Foundations of Mental Simulation: Future Prediction of Latent Representations on Dynamic Scenes

Aran Nayebi, Rishi Rajalingham, Mehrdad Jazayeri, Guangyu Robert Yang

NeurIPS 2023 · Probabilistic and scientific modeling › Dynamical systems · approach: Latent-space control and planning · 3 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Future-predict in video foundation model latents using LSTM or CTRNN dynamicsVC-1 and R3M latent models · Mental-Pongneural predictivity: 19.03% and 24.79% -> 46.34-48.83%Improved
Use dynamic video-pretrained latents rather than static-image-pretrained latentsLatent future prediction models · Mental-Pongneural predictivity: at most 29.67% -> 46.34-48.83%Improved
Use VC-1 or R3M representations instead of VIP representationsVideo foundation model latent predictors · Mental-Pongneural predictivity: 7.37%-18.12% -> 46.34-48.83%Improved
Add a dynamics module to the VC-1 or R3M encoder latents (ablation)VC-1 and R3M · Mental-Pongneural predictivity: 19.03% and 24.79% -> 46.34-48.83%Improved
Increase SVG input resolution from 64×64 to 128×128 (ablation)SVG · Physion OCP held-out scenesaccuracyNo clear effect
Add temporal RandAugment (ablation)FitVid · Physion OCP held-out scenesaccuracyNo clear effect
Use CTRNN dynamics instead of LSTM dynamics (ablation)VC-1 and R3M latent models · Mental-Pongneural predictivityImproved
Pretrain dynamics on Kinetics-700 instead of Physion (ablation)VIP, VC-1, and R3M latent models · Physion OCPaccuracyNo clear effect
Pretrain dynamics on Kinetics-700 instead of Physion (ablation)VIP, VC-1, and R3M latent models · Mental-Pongneural predictivityNo clear effect
Use object-slot C-SWM instead of pixel-wise or latent predictionC-SWM · Physion OCPaccuracy and human judgment probability matchHurt
Use end-to-end pixel-wise prediction instead of latent video-model predictionFitVid and SVG · Mental-Pongneural predictivity: 14.44%-25.35% vs. 46.34-48.83%Hurt
Use end-to-end pixel-wise prediction rather than latent video-model predictionFitVid and SVG · Physion OCP and Mental-Ponggeneralization across environments: 61.12%-63.31%; 14.44%-25.35%Mixed

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy