Research map / Vision-language and multimodal / Human and embodied modeling
MEIcoder: Decoding Visual Stimuli from Neural Activity by Leveraging Most Exciting Inputs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MEIcoder, an end-to-end decoder that reconstructs visual stimuli from V1 population activity by using neuron-specific most exciting inputs as a readin prior, SSIM loss, and adversarial training, and releases a multi-dataset decoding benchmark.
Problem
Decoding natural images from neural population activity in V1 is hard when single-subject data is scarce and few neurons are recorded; prior GenAI-based decoders hallucinate and direct decoders overfit.
Approach
Inject neuron-specific most exciting inputs (MEIs) as a computational prior in a readin module that weights MEIs by learned neuron context representations, feed the result into a shared CNN core, and train with a negative log-SSIM loss plus an adversarial discriminator loss.
Key results
- On Brainreader, MEIcoder reaches SSIM .400, PixCorr .679, Alex(5) .990, outperforming the best baseline InvEnc (SSIM .321, Alex(5) .896).
- On SENSORIUM 2022, MEIcoder (FT) reaches Alex(5) .908 and SSIM .318 versus MindEye2 Alex(5) .762.
- MEIcoder exceeds 95% Alex(2) with around 1,000 neurons and reconstructs digits with around 2,500 neurons.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MEIs in the readin instead of removing neural characterization (ablation) | MEIcoder · three-dataset average | reconstruction quality across all reported metrics | Helped |
| include neuron embeddings in the readin input (ablation) | MEIcoder · three-dataset average | reconstruction quality | Helped |
| use SSIM-based reconstruction loss instead of MSE (ablation) | MEIcoder · three-dataset average | perceptual reconstruction quality | Helped |
| replace MEIs with gradient-based linear receptive fields (ablation) | MEIcoder · Brainreader test | SSIM: .400 -> .364 (-9%) | Hurt |
| replace MEIs with gradient-based linear receptive fields (ablation) | MEIcoder · Brainreader test | PixCorr: .679 -> .663 (-2.4%) | Hurt |
| add Gaussian noise to MEIs at increasing standard deviations (ablation) | MEIcoder · Brainreader test | SSIM: .400 -> .402; .390; .332; .287 | Mixed |
| use MEIcoder instead of InvEnc | visual stimulus decoding · Brainreader test | SSIM: .321 -> .400 | Helped |
| use MEIcoder instead of CAE | visual stimulus decoding · SENSORIUM 2022 test | PixCorr: .539 -> .503 | Hurt |
| use MEIcoder instead of InvEnc | visual stimulus decoding · Synthetic Cat V1 test | SSIM: .771 -> .774 | Helped |
Surprising findings
- Removing MEIs caused the largest performance drop in ablations, larger than removing the SSIM loss or neuron embeddings.
- On the Synthetic Cat V1 dataset, InvEnc outperformed MEIcoder on SSIM, PixCorr, and Alex metrics.
Limitations
- Only tested on V1 data; higher visual areas such as V4 not evaluated.
- Transfer from multi-subject pre-training helped on Brainreader but not on SENSORIUM 2022.
- Evaluation metrics such as PixCorr and two-way identification plateau while visual quality keeps improving.
Experimental setup
Datasets: Brainreader SENSORIUM 2022 Synthetic Cat V1 ImageNet
Benchmarks: Brainreader SENSORIUM 2022 Synthetic Cat V1
Metrics: SSIM PixCorr Alex(2) Alex(5) Inception two-way identification CLIP two-way identification EfficientNet-B1 correlation distance SwAV correlation distance
Baselines: InvEnc EGG MonkeySee CAE MindEye2 MindEye2 (FT)
Builds on: most exciting inputs SSIM InvEnc MonkeySee Energy Guided Diffusion
Closest papers by meaning
- Brain decoding: toward real-time reconstruction of visual perception (ICLR 2024 · similarity 0.88)
- MonkeySee: Space-time-resolved reconstructions of natural images from macaque multi-unit activity (NeurIPS 2024 · similarity 0.87)
- ZEBRA: Towards Zero-Shot Cross-Subject Generalization for Universal Brain Visual Decoding (NeurIPS 2025 · similarity 0.87)
- Neural Encoding and Decoding at Scale (ICML 2025 · similarity 0.87)
- Toward Generalizing Visual Brain Decoding to Unseen Subjects (ICLR 2025 · similarity 0.86)
- SynBrain: Enhancing Visual-to-fMRI Synthesis via Probabilistic Representation Learning (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.