Research map / Vision-language and multimodal / Cross-modal learning
Multi-Modal Latent Variables for Cross-Individual Primary Visual Cortex Modeling and Analysis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes miVAE, a multi-modal identifiable VAE with two-level latent disentanglement that aligns mouse V1 neural activity and visual stimuli in a shared latent space, plus a score-based attribution method to trace latents back to neurons and stimulus regions.
Problem
Modeling V1 requires integrating partial calcium-imaging recordings with visual stimuli, and cross-individual variability in neuron populations and firing patterns prevents models from generalizing across mice without subject-specific fine-tuning.
Approach
A multi-modal identifiable VAE splits neural and stimulus latents into idiosyncratic/preserved and stimulus-related/irrelevant parts, uses bidirectional (activity-to-stimulus and stimulus-to-activity) generative modeling with shared latent zc, and attributes latent variables to original neurons and stimulus pixels via a score function.
Key results
- miVAE reaches latent encoding correlation 0.8694 (linear) and 0.9149 (nonlinear) in stage 1 vs. best baseline MVAE/iVAE, and 0.9094 nonlinear decoding.
- On cross-mouse pairs with identical stimuli, miVAE achieves 0.8984 linear latent correlation vs. 0.7449 for iVAE and 0.7316 for MVAE.
- Important neuron subpopulation (~700 neurons) yields 91.29% stimulus classification accuracy, exceeding the full population (87.24%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use miVAE instead of the strongest reported baseline | Latent encoding and decoding · Sensorium 2023 unseen mice | Pearson correlation: 0.8692 -> 0.9094 | Helped |
| Combine LCM with LMM (ablation) | miVAE · Sensorium 2023 unseen mice | Pearson correlation | Helped |
| Use LCM alone instead of LMM alone (ablation) | miVAE · Sensorium 2023 unseen mice | Pearson correlation | Helped |
| Include the idiosyncratic neural latent variable zx_m (ablation) | miVAE · Sensorium 2023 unseen mice | Pearson correlation | Helped |
| Include the activity-irrelevant stimulus latent variable zy_m (ablation) | miVAE · Sensorium 2023 unseen mice | Pearson correlation | No clear effect |
| Set latent size to 16 (ablation) | miVAE · Sensorium 2023 unseen mice | Generalization error | Helped |
| Increase the number of training mice (ablation) | miVAE · Cross-individual generalization | Pearson correlation | Helped |
| Increase the number of training trials (ablation) | miVAE · Cross-individual generalization | Pearson correlation | Helped |
| Select important neurons using attribution weights rather than use the full population | Stimulus classification · Three validation mice | Classification accuracy: 87.24% -> 91.29% | Helped |
Surprising findings
- Retaining zx_m while omitting zy_m maintains robust performance, suggesting the stimulus-irrelevant latent contributes little.
Limitations
- Stimulus-irrelevant latent zy_m uses a standard normal prior without receptive-field information, limiting its modeling.
- Evaluation is limited to a single dataset (Sensorium 2023) of mouse V1 calcium imaging.
- Direct comparison between original-data-space models and latent-space coding is not possible, per the authors.
Experimental setup
Datasets: Sensorium 2023
Benchmarks: Sensorium 2023
Metrics: single-trial Pearson correlation classification accuracy
Baselines: V1-FM VAE iVAE MVAE MMVAE MoPoE MEME MMVE
Builds on: iVAE MMVE pi-VAE V1-FM
Closest papers by meaning
- Multi-modal Gaussian Process Variational Autoencoders for Neural and Behavioral Data (ICLR 2024 · similarity 0.91)
- Modeling Dynamic Neural Activity by combining Naturalistic Video Stimuli and Stimulus-independent Latent Factors (NeurIPS 2025 · similarity 0.88)
- Time-Evolving Dynamical System for Learning Latent Representations of Mouse Visual Neural Activity (NeurIPS 2025 · similarity 0.88)
- Identifying interactions across brain areas while accounting for individual-neuron dynamics with a Transformer-based variational autoencoder (NeurIPS 2025 · similarity 0.87)
- Poisson Variational Autoencoder (NeurIPS 2024 · similarity 0.87)
- Multi-modal brain encoding models for multi-modal stimuli (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.