Research map / Architectures and efficiency / CNN design
MonkeySee: Space-time-resolved reconstructions of natural images from macaque multi-unit activity
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Reconstructs THINGS natural images from macaque V1/V4/IT multi-unit activity with a fully convolutional homeomorphic decoder, adding space-time-resolved occlusion analysis and a learned inverse retinotopic mapping layer.
Problem
Decoding naturalistic images from macaque multi-unit activity across visual areas V1, V4 and IT, and understanding how spatial and temporal information in neural signals maps to reconstructed pixels.
Approach
A fully convolutional decoder trained from scratch maps MUA from 576 electrodes into 2D retinal embeddings (fixed via pre-trained Inception features or learned Gaussian receptive fields) and then a U-Net reconstructs images, with spatial and spatiotemporal occlusion analyses to probe area and time-window contributions.
Key results
- Spatiotemporal decoder achieved highest AlexNet feature correlation in most layers, e.g. FC8 0.610 vs 0.446 for the baseline and FC7 0.579 vs 0.434.
- Spatial, spatiotemporal and end-to-end decoders all outperformed the baseline qualitatively and quantitatively across AlexNet layers conv1 to fc8.
- Learned receptive fields grow with eccentricity, with larger fields in the periphery and smaller ones centrally.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use spatiotemporal rather than baseline decoding | MonkeySee decoder · THINGS test images | AlexNet FC8 feature correlation: 0.446 -> 0.610 | Helped |
| Use spatiotemporal rather than spatial decoding | MonkeySee decoder · THINGS test images | AlexNet FC8 feature correlation: 0.579 -> 0.610 | Helped |
| Use spatiotemporal rather than end-to-end decoding | MonkeySee decoder · THINGS test images | AlexNet FC8 feature correlation: 0.543 -> 0.610 | Helped |
| Use spatiotemporal rather than baseline decoding | MonkeySee decoder · THINGS test images | AlexNet FC7 feature correlation: 0.434 -> 0.579 | Helped |
| Use spatiotemporal rather than spatial decoding | MonkeySee decoder · THINGS test images | AlexNet FC7 feature correlation: 0.534 -> 0.579 | Helped |
| Use spatiotemporal rather than end-to-end decoding | MonkeySee decoder · THINGS test images | AlexNet FC7 feature correlation: 0.541 -> 0.579 | Helped |
| Use spatial decoding rather than the baseline decoder | MonkeySee decoder · THINGS test images | AlexNet FC8 feature correlation: 0.446 -> 0.579 | Helped |
| Use end-to-end decoding rather than the baseline decoder | MonkeySee decoder · THINGS test images | AlexNet FC8 feature correlation: 0.446 -> 0.543 | Helped |
Surprising findings
- V1-constrained reconstructions had the highest colorfulness, while IT reconstructions correlated more with deeper AlexNet layers.
Limitations
- No error bars reported for the quantitative results.
- Evaluated on a single macaque and a small 100-trial test set.
- Applicability to non-invasive fMRI or human intracranial recordings is limited.
Experimental setup
Datasets: THINGS THINGS macaque visual cortex MUA recordings (in-house)
Benchmarks: THINGS macaque MUA test set (100 trials)
Metrics: Pearson feature correlation across AlexNet layers colorfulness (Hasler and Süsstrunk)
Baselines: Baseline decoder (modified U-Net expansive path, after Shen et al. 2019)
Builds on: U-Net pix2pix-style adversarial training Le et al. 2022 fMRI image reconstruction Shen et al. 2019 end-to-end reconstruction
Closest papers by meaning
- Brain decoding: toward real-time reconstruction of visual perception (ICLR 2024 · similarity 0.88)
- MEIcoder: Decoding Visual Stimuli from Neural Activity by Leveraging Most Exciting Inputs (NeurIPS 2025 · similarity 0.87)
- Toward Generalizing Visual Brain Decoding to Unseen Subjects (ICLR 2025 · similarity 0.87)
- See Through Their Minds: Learning Transferable Brain Decoding Models from Cross-Subject fMRI (AAAI 2025 · similarity 0.87)
- Transformer brain encoders explain human high-level visual responses (NeurIPS 2025 · similarity 0.87)
- NSD-Imagery: A Benchmark Dataset for Extending fMRI Vision Decoding Methods to Mental Imagery (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.