Research map / Video / Video segmentation and tracking
NeuralFlix: A Simple While Effective Framework for Semantic Decoding of Videos from Non-invasive Brain Recordings
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NeuralFlix, a two-phase framework that trains a contrastively augmented fMRI encoder aligned to CLIP embeddings and uses it to condition a video diffusion model (Stable Diffusion with temporal layers) with dependent prior noise to reconstruct videos from fMRI.
Problem
Reconstructing dynamic visual experiences (videos) from non-invasive fMRI is hard due to noise, spatial redundancy, and hemodynamic temporal lag, and video decoding from fMRI has been little explored compared with static images.
Approach
Spatial masking and weighted temporal interpolation are used as augmentations for contrastive fMRI representation learning aligned to CLIP text/image embeddings, and the diffusion model's initial noise is replaced by dependent noise (shared plus per-frame components) to compensate for low fMRI SNR and to exploit correlated fMRI signals across time.
Key results
- SSIM of 0.225, 0.224, and 0.211 for Subjects 1-3, improving over prior SOTA (Chen, Qing, and Zhou 2023) by 20.97%, 31.00%, and 12.30% respectively.
- Dependent noise with ratio sqrt(beta)=0.25 raised SSIM on Subject 1 validation from 0.204 to 0.242 in the ablation study.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add spatial masking with ratio 0.2 (ablation) | NeuralFlix · Validation set | SSIM: 0.171 -> 0.204 | Helped |
| Increase spatial mask ratio from 0.2 to 0.4 (ablation) | NeuralFlix · Validation set | SSIM: 0.204 -> 0.184 | Hurt |
| Increase spatial mask ratio from 0.2 to 0.6 (ablation) | NeuralFlix · Validation set | SSIM: 0.204 -> 0.184 | Hurt |
| Increase temporal interpolation ratio from 1/3 to 1/2 (ablation) | NeuralFlix · Validation set | SSIM: 0.204 -> 0.186 | Hurt |
| Set spatial and temporal augmentation loss weights to 0.5 instead of 1 (ablation) | NeuralFlix · Validation set | SSIM: 0.204 -> 0.191 | Hurt |
| Set spatial and temporal loss weights to 0.25 and 0.75 (ablation) | NeuralFlix · Validation set | SSIM: 0.204 -> 0.186 | Hurt |
| Set spatial and temporal loss weights to 0.75 and 0.25 (ablation) | NeuralFlix · Validation set | SSIM: 0.204 -> 0.185 | Hurt |
| Add dependent noise with ratio 0.25 (ablation) | NeuralFlix · Validation set | SSIM: 0.204 -> 0.242 | Helped |
| Add dependent noise with ratio 0.5 (ablation) | NeuralFlix · Validation set | SSIM: 0.204 -> 0.230 | Helped |
| Add dependent noise with ratio 0.75 (ablation) | NeuralFlix · Validation set | SSIM: 0.204 -> 0.225 | Helped |
| Replace the fMRI sequence with time-averaged brain signals (ablation) | NeuralFlix · Subject 2 | 50-way-top-1 video classification accuracy: 0.187 ± 0.015 -> 0.157 ± 0.013 | Hurt |
| Replace the fMRI sequence with time-averaged brain signals (ablation) | NeuralFlix · Subject 3 | 50-way-top-1 video classification accuracy: 0.195 ± 0.014 -> 0.158 ± 0.012 | Hurt |
Surprising findings
- Time-averaging brain signals reduced video classification accuracy (Subject 2: 0.187 to 0.157), which the authors take as evidence the model uses fMRI dynamics rather than only diffusion priors.
Limitations
- Semantic categories of training and test clips overlap only 0.56 (IoU), limiting generalization.
- Evaluated only on one public fMRI-video dataset with three subjects.
- Video windows are short (2-second segments at 3 FPS), extended only as GPU memory allows.
Experimental setup
Datasets: Wen et al. 2018 fMRI-video dataset HCP (pre-training of fMRI encoder)
Benchmarks: Wen et al. 2018 fMRI-video dataset (3 subjects)
Metrics: SSIM 50-way top-1 image classification accuracy 50-way top-1 video classification accuracy
Baselines: Chen, Qing, and Zhou 2023 Kupershmidt et al. 2022 Wang et al. 2022 Wen et al. 2018
Builds on: Stable Diffusion DC-MAE Chen et al. 2023 video decoding InfoNCE CLIP
Closest papers by meaning
- Neuro-Vision to Language: Enhancing Brain Recording-based Visual Reconstruction and Language Interaction (NeurIPS 2024 · similarity 0.88)
- Brain decoding: toward real-time reconstruction of visual perception (ICLR 2024 · similarity 0.87)
- MoRE-Brain: Routed Mixture of Experts for Interpretable and Generalizable Cross-Subject fMRI Visual Decoding (NeurIPS 2025 · similarity 0.87)
- SIM: Surface-based fMRI Analysis for Inter-Subject Multimodal Decoding from Movie-Watching Experiments (ICLR 2025 · similarity 0.87)
- Brain Harmony: A Multimodal Foundation Model Unifying Morphology and Function into 1D Tokens (NeurIPS 2025 · similarity 0.87)
- NeuroClips: Towards High-fidelity and Smooth fMRI-to-Video Reconstruction (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.