Research map / Video / Video segmentation and tracking
Reanimating Images using Neural Representations of Dynamic Stimuli
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Decodes fine-grained optical flow from fMRI responses to dynamic videos using DINOv2 image features and a classifier over quantized flow, then animates the initial frame with DragNUWA; also benchmarks video vs image encoders for brain prediction.
Problem
Decoding and understanding the brain's representation of motion in dynamic visual stimuli from fMRI, which prior brain-to-video work entangles with static appearance; also improving computer vision motion prediction using neural data.
Approach
Disentangle static appearance from motion by conditioning on DINOv2 features of the initial frame plus fMRI to classify quantized optical flow, then use a motion-conditioned video diffusion model (DragNUWA) to reanimate the frame; apply the flow decoder to frames decoded by MindVideo for full video decoding.
Key results
- Motion decoded from fMRI achieves thresholded average EPE of 0.543, 0.572, 0.634 for subjects 1-3 vs 1.192 for SVD (Best), all p<0.001.
- VideoMAE Large predicts fMRI better than best static image model: Pearson r 0.285/0.314/0.324 vs CLIP ConvNeXt 0.219/0.263/0.272.
- Applying the method to MindVideo initial frames improves video generation metrics (e.g., VideoMAE CosSim 0.762-0.774 vs 0.742 for MindVideo).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use BrainNRDS motion prediction conditioned on fMRI rather than SVD motion prediction without neural data | BrainNRDS · Dynamic Natural Vision test, Subjects 1–3 | thresholded average EPE: S1: 1.192 -> 0.543; S2: 1.192 -> 0.572; S3: 1.192 -> 0.634 | Helped |
| Use BrainNRDS end-to-end video generation instead of MindVideo | Dynamic Natural Vision test · Dynamic Natural Vision test | VideoMAE cosine similarity: 0.742±0.006 -> 0.769±0.006 | Helped |
| Use BrainNRDS end-to-end video generation instead of MindVideo | Dynamic Natural Vision test · Dynamic Natural Vision test | CLIP cosine similarity: 0.879±0.004 -> 0.896±0.003 | Helped |
| Use BrainNRDS end-to-end video generation instead of MindVideo | Dynamic Natural Vision test · Dynamic Natural Vision test | SSIM: 0.171±0.02 -> 0.214±0.01 | Helped |
| Use VideoMAE Large video features instead of CLIP ConvNeXt image features | Voxel-wise fMRI encoding · Dynamic Natural Vision, Subjects 1–3 | Pearson r: S1: 0.219 -> 0.285; S2: 0.263 -> 0.314; S3: 0.272 -> 0.324 | Helped |
| Use VC-1 embodied-AI features instead of CLIP ConvNeXt image features | Voxel-wise fMRI encoding · Dynamic Natural Vision, Subjects 1–3 | Pearson r: S1: 0.219 -> 0.260; S2: 0.263 -> 0.290; S3: 0.272 -> 0.294 | Helped |
Surprising findings
- Averaging embeddings across frames from temporally unaligned encoders does not improve brain prediction performance.
Limitations
- Generalizability to other dataset distributions is unclear; evaluated on a single dataset.
- Subject-specific training; cross-subject alignment not explored.
- fMRI's slow temporal resolution limits fine-grained motion decoding.
Experimental setup
Datasets: Dynamic Natural Vision dataset
Benchmarks: Dynamic Natural Vision dataset
Metrics: end point error Pearson correlation VideoMAE cosine similarity CLIP cosine similarity SSIM
Baselines: MindVideo Stable Video Diffusion ResNet-50 CLIP ConvNeXt DINOv2 VIP VC-1 R3M
Builds on: MindVideo DragNUWA RAFT FlowSAM
Closest papers by meaning
- EEG2Video: Towards Decoding Dynamic Visual Perception from EEG Signals (NeurIPS 2024 · similarity 0.88)
- Brain decoding: toward real-time reconstruction of visual perception (ICLR 2024 · similarity 0.88)
- NEvo: Neural-Guided Evolutionary Video Synthesis for Dynamic Visual Selectivity (NeurIPS 2026 · similarity 0.87)
- SIM: Surface-based fMRI Analysis for Inter-Subject Multimodal Decoding from Movie-Watching Experiments (ICLR 2025 · similarity 0.87)
- Time-Evolving Dynamical System for Learning Latent Representations of Mouse Visual Neural Activity (NeurIPS 2025 · similarity 0.86)
- Flow Snapshot Neurons in Action: Deep Neural Networks Generalize to Biological Motion Perception (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.