Research map / Image generation / Autoregressive and latent generation
Brain decoding: toward real-time reconstruction of visual perception
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Decodes and generates natural images from MEG signals on THINGS-MEG using a convolutional brain module that aligns MEG to pretrained image embeddings, followed by retrieval or latent diffusion generation.
Problem
fMRI-based visual reconstruction is limited by slow temporal resolution (~0.5 Hz), preventing real-time decoding; it is unknown whether MEG can generate open sets of natural images with high temporal resolution.
Approach
Train a dilated residual ConvNet brain module to map MEG windows to pretrained image latents (CLIP, AutoKL) using a combined CLIP contrastive and MSE loss, then condition a latent diffusion model on the predicted latents to generate images, with sliding and growing windows for temporal analysis.
Key results
- Deep ConvNet brain module yields about 7X improvement in retrieval over linear Ridge baselines; VGG-19, CLIP-Vision and DINOv2 reach top-5 accuracies of 70.33%, 68.66% and 68.00% on the small test set (chance 2.5%).
- Time-resolved retrieval peaks a few hundred ms after both image onset and offset, with category-level information dominating as early as 250 ms.
- MEG full-window generations on THINGS-MEG preserve high-level category information but retain fewer low-level features than fMRI-based reconstructions on NSD (e.g., PixCorr 0.090 vs 0.305).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the deep ConvNet decoder instead of linear Ridge regression | MEG image retrieval · THINGS-MEG | retrieval performance: 7X | Helped |
| Average predictions across trials and subjects rather than use no average (ablation) | MEG image generation · THINGS-MEG | CLIP: 0.700 -> 0.811 | Helped |
| Average predictions across trials and subjects rather than use no average (ablation) | MEG image generation · THINGS-MEG | PixCorr: 0.058 ± 0.011 -> 0.090 ± 0.009 | Helped |
| Use VGG-19 classifier_dropout_5 features instead of the last layer (ablation) | MEG retrieval with VGG-19 · THINGS-MEG small test set | top-5 accuracy: 70.333 -> 74.500 | Helped |
| Use VGG-19 avgpool features instead of the last layer (ablation) | MEG retrieval with VGG-19 · THINGS-MEG large test set | top-5 accuracy: 12.292 -> 17.417 | Helped |
| Use VGG-19 maxpool2d_35 features instead of the last layer (ablation) | MEG retrieval with VGG-19 · THINGS-MEG small test set | top-5 accuracy: 70.333 -> 64.333 | Hurt |
| Use fMRI betas rather than MEG signals | Image generation pipeline · NSD vs THINGS-MEG | PixCorr: 0.076 ± 0.005 -> 0.305 ± 0.007 | Helped |
Surprising findings
- Low-level MEG features are most decodable from windows immediately around image onset and offset rather than during sustained presentation.
- Decoding performance rises after image offset, suggesting offset responses carry both low- and high-level information.
Limitations
- MEG generations preserve fewer low-level features than 7T fMRI, consistent with lower spatial resolution of MEG.
- Approach depends on pretrained frozen embeddings and does not fine-tune image or generation modules.
- Extension to EEG, a more clinically common modality, is left for future work.
Experimental setup
Datasets: THINGS-MEG THINGS Natural Scenes Dataset (NSD)
Benchmarks: THINGS-MEG small test set THINGS-MEG large test set NSD
Metrics: top-5 accuracy relative median rank PixCorr SSIM AlexNet(2) AlexNet(5) Inception CLIP SwAV
Baselines: Ridge regression linear decoder fMRI NSD reconstruction pipeline (Ridge regression)
Builds on: CLIP Ridge regression encoding/decoding (Défossez et al. brain module) Stable Diffusion image-to-image pipeline (Takagi & Nishimoto style fMRI reconstr VGG-19
Closest papers by meaning
- NSD-Imagery: A Benchmark Dataset for Extending fMRI Vision Decoding Methods to Mental Imagery (CVPR 2025 · similarity 0.92)
- Visual Decoding and Reconstruction via EEG Embeddings with Guided Diffusion (NeurIPS 2024 · similarity 0.91)
- See Through Their Minds: Learning Transferable Brain Decoding Models from Cross-Subject fMRI (AAAI 2025 · similarity 0.91)
- Human-Aligned Image Models Improve Visual Decoding from the Brain (ICML 2025 · similarity 0.91)
- Toward Generalizing Visual Brain Decoding to Unseen Subjects (ICLR 2025 · similarity 0.90)
- EEG2Video: Towards Decoding Dynamic Visual Perception from EEG Signals (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.