Research map / Probabilistic and scientific modeling / Dynamical systems
Multi-modal Gaussian Process Variational Autoencoders for Neural and Behavioral Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MM-GPVAE, a multi-modal GP-VAE that jointly models Poisson neural spike counts and a second behavioral/stimulus modality with shared and independent latents, parameterized in a pruned Fourier basis.
Problem
Identifying temporally smooth latent structure shared across and specific to simultaneously recorded neural and behavioral/stimulus modalities, which existing LVMs handle poorly when they lack temporal structure, interpretability, or flexible decoders.
Approach
Combines GPFA-style linear neural decoder (Poisson likelihood) with a GP-VAE deep decoder for the second modality, with latents partitioned into shared and independent subspaces via a loadings matrix, and GP latents parameterized in a pruned Fourier domain for diagonal prior covariance and better latent identification.
Key results
- On simulated data, MM-GPVAE reconstructs held-out images better than a single-modality GP-VAE trained on images alone, and recovers shared latents more accurately than unimodal GPFA/GP-VAE.
- Fourier-domain GP-VAE shows much smoother latents and better latent identification than standard time-domain GP-VAE on a rotating MNIST simulation.
- On the hawkmoth data, removing the shared latent reduces image reconstruction accuracy by roughly 70% and rate reconstruction by about 5%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use a Fourier-domain representation instead of the standard GP-VAE representation | GP-VAE · Simulated rotating MNIST | latent identification accuracy | Helped |
| jointly model images and neural spikes instead of fitting an image-only GP-VAE | MM-GPVAE · Simulated MNIST and neural spikes | image reconstruction MSE | Helped |
| include image data alongside neural activity | MM-GPVAE · Simulated MNIST and neural spikes | neural rate estimation accuracy | Helped |
| use both modalities instead of either modality alone | MM-GPVAE · Simulated shared latent | shared latent identification accuracy | Helped |
| prune high Fourier frequencies (ablation) | MM-GPVAE · Simulated shared and independent latents | latent identification accuracy | Helped |
| increase the number of neurons (ablation) | MM-GPVAE · Simulated neural and shared latents | neural latent identification accuracy | Helped |
| increase the number of neurons (ablation) | MM-GPVAE · Simulated shared latent | shared latent identification accuracy | Helped |
| increase the number of neurons (ablation) | MM-GPVAE · Simulated image latent | image latent identification accuracy | No clear effect |
| encode both modalities rather than only the model's own modality (ablation) | Unimodal GP-VAE and GPFA comparisons · Simulated data | model performance | No clear effect |
| remove the GP prior over time (ablation) | MM-VAE · Simulated occluded MNIST | reconstruction of the true image | Hurt |
| remove the shared latent (ablation) | MM-GPVAE · Hawkmoth visual stimulus | reconstruction accuracy: about a 70% reduction | Hurt |
| remove the shared latent (ablation) | MM-GPVAE · Hawkmoth neural rates | reconstruction accuracy: about a 5% reduction | Hurt |
Surprising findings
- Encoding both modalities for unimodal GP-VAE or GPFA had little effect on their performance, so the MM-GPVAE gains come from the model structure rather than the encoder.
Limitations
- Latent identification in the time-domain GP-VAE fails for long trials without Fourier pruning, and the approach depends on trial length and hyperparameter choices.
- Latent dimensionality and shared-dimension selection are heuristic and cross-validation variance was high.
- TNDM and PSID comparisons are limited to a simulated setting and a single primate reaching dataset.
Experimental setup
Datasets: simulated rotating/scaling MNIST digits with Poisson spike counts Drosophila whole-brain calcium imaging with DeepLabCut limb positions Manduca sexta spike trains from 10 wing muscles with visual stimulus primate reaching data
Benchmarks: simulated MNIST and Poisson spikes fly calcium imaging and limb positions hawkmoth EMG and visual stimulus primate reaching
Metrics: mean squared error latent identification accuracy
Baselines: GP-VAE GPFA TNDM PSID standard VAE MM-VAE
Builds on: GPFA GP-VAE PSID TNDM
Closest papers by meaning
- Multi-Modal Latent Variables for Cross-Individual Primary Visual Cortex Modeling and Analysis (AAAI 2025 · similarity 0.91)
- Conditionally-Conjugate Gaussian Process Factor Analysis for Spike Count Data via Data Augmentation (ICML 2024 · similarity 0.89)
- Identifying interactions across brain areas while accounting for individual-neuron dynamics with a Transformer-based variational autoencoder (NeurIPS 2025 · similarity 0.88)
- Dynamical modeling of nonlinear latent factors in multiscale neural activity with real-time inference (NeurIPS 2025 · similarity 0.88)
- Poisson Variational Autoencoder (NeurIPS 2024 · similarity 0.88)
- Modeling Dynamic Neural Activity by combining Naturalistic Video Stimuli and Stimulus-independent Latent Factors (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.