Research map / Video / Video generation
NeuroClips: Towards High-fidelity and Smooth fMRI-to-Video Reconstruction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NeuroClips, a fMRI-to-video reconstruction framework that combines a perception reconstructor (blurry low-level video) and a semantics reconstructor (keyframe) to condition AnimateDiff, reaching up to 6s video at 8FPS via multi-fMRI fusion.
Problem
Reconstructing smooth, high-fidelity videos from fMRI is hard because fMRI has low temporal resolution (2s) and low SNR, and prior methods (e.g., MinD-Video) lack low-level perceptual control and produce non-smooth, semantically inaccurate videos.
Approach
Decode two complementary signals from fMRI: a blurry low-level video (perception reconstructor, aligned to VAE latents of downsampled frames with temporal upsampling attention) and a high-level keyframe (semantics reconstructor, aligned to OpenCLIP ViT-bigG/14 with BiMixCo, diffusion prior, and BLIP-2 text alignment), then inject both into a frozen pretrained T2V diffusion model (AnimateDiff) at i
Key results
- NeuroClips achieves SSIM 0.390 vs 0.171 for MinD-Video (a 128% improvement) and CLIP-pcc 0.738 vs 0.408 on cc2017 (3FPS downsampled comparison).
- 50-way video semantic accuracy 0.220 for NeuroClips vs 0.197 for MinD-Video; frame-level 50-way 0.203 vs 0.174.
- Multi-fMRI fusion enables continuous video reconstruction up to 6s at 8FPS without additional training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add NeuroClips rather than MinD-Video | fMRI-to-video reconstruction · cc2017 test set | CLIP-pcc: 0.408 -> 0.738 | Helped |
| add NeuroClips rather than MinD-Video | fMRI-to-video reconstruction · cc2017 test set | SSIM: 0.171 -> 0.390 | Helped |
| add NeuroClips rather than MinD-Video | fMRI-to-video reconstruction · cc2017 test set | video 2-way accuracy: 0.839 -> 0.834 | Hurt |
| add NeuroClips rather than MinD-Video | fMRI-to-video reconstruction · cc2017 test set | video 50-way accuracy: 0.197 -> 0.220 | Helped |
| add keyframe and keyframe captioning (ablation) | NeuroClips · cc2017 subject 1 | frame 2-way accuracy: 0.702 -> 0.799 | Helped |
| add keyframe and keyframe captioning (ablation) | NeuroClips · cc2017 subject 1 | SSIM: 0.413 -> 0.392 | Hurt |
| add blurry clip guidance (ablation) | NeuroClips · cc2017 subject 1 | SSIM: 0.256 -> 0.392 | Helped |
| add blurry clip guidance (ablation) | NeuroClips · cc2017 subject 1 | video 50-way accuracy: 0.213 -> 0.208 | Hurt |
| replace BLIP-2 keyframe captioning with GIT captioning (ablation) | NeuroClips · cc2017 subject 1 | frame 2-way accuracy: 0.799 -> 0.785 | Hurt |
| replace BLIP-2 keyframe captioning with GIT captioning (ablation) | NeuroClips · cc2017 subject 1 | CLIP-pcc: 0.736 -> 0.728 | Hurt |
| fine-tune the projector (ablation) | text-image alignment on MSCOCO 2017 · MSCOCO 2017 | Image2Text: 81.3% -> 95.2% | Helped |
| fine-tune the projector (ablation) | text-image alignment on MSCOCO 2017 · MSCOCO 2017 | Text2Image: 78.7% -> 95.2% | Helped |
Surprising findings
- Removing the blurry video branch improves semantic metrics (2-way 0.838 vs 0.830) while removing keyframes improves SSIM (0.413 vs 0.392), revealing a trade-off between perceptual and semantic reconstruction.
- Keyframe retrieval accuracy averages only about 22% on cc2017, far below image-fMRI retrieval reported in prior work, attributed to many test categories absent from training.
Limitations
- Framework is bulky and relies on extending keyframes into the reconstructed video, as no model reconstructs video directly from CLIP latent space.
- Does not reconstruct cross-scene fMRI well (fMRI recorded during clip switching).
- Evaluated on a single public dataset with only three subjects; semantic accuracy limited by out-of-distribution test categories.
Experimental setup
Datasets: cc2017 fMRI-video dataset MSCOCO 2017 ImageNet Kinetics-400
Benchmarks: cc2017 fMRI-video dataset
Metrics: SSIM PSNR N-way top-K classification accuracy (2-way, 50-way) CLIP-pcc
Baselines: Nishimoto Wen Wang Kupershmidt MinD-Video
Builds on: MinD-Video DALLE·2 AnimateDiff ControlNet MixCo
Closest papers by meaning
- FlatClip: Reusing Image Foundation Models for fMRI Representation Learning via Cortical Flatmaps (NeurIPS 2026 · similarity 0.88)
- MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data (ICML 2024 · similarity 0.88)
- Brain decoding: toward real-time reconstruction of visual perception (ICLR 2024 · similarity 0.87)
- SIM: Surface-based fMRI Analysis for Inter-Subject Multimodal Decoding from Movie-Watching Experiments (ICLR 2025 · similarity 0.87)
- MindBridge: A Cross-Subject Brain Decoding Framework (CVPR 2024 · similarity 0.87)
- NeuralFlix: A Simple While Effective Framework for Semantic Decoding of Videos from Non-invasive Brain Recordings (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.