Research map / Time series / Causal and generative time series
Integrating Multimodal Data for Joint Generative Modeling of Complex Dynamics
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Multimodal Teacher Forcing (MTF), which uses a multimodal VAE to produce a sparse teacher-forcing signal that guides training of a dendPLRNN dynamical systems reconstruction model from mixed Gaussian, ordinal, count and categorical data.
Problem
Reconstructing nonlinear dynamical systems from multimodal time series with discrete (ordinal, count, categorical) observations, where standard teacher forcing requires invertible decoders and fails for discrete data and chaotic dynamics.
Approach
Use a multimodal VAE, whose decoders are shared with the reconstruction model and whose prior is given by the dendPLRNN dynamics, to generate a sparse teacher-forcing control signal in a common latent space, replacing the pseudo-inverse of decoders used in STF/GTF.
Key results
- On simulated Lorenz, Rossler and Lewis-Glass data, MTF achieved lower D_stsp than SVAE, BPTT and other baselines (e.g. Lorenz D_stsp 3.4 vs 11.1 for SVAE).
- MTF reconstructed the Lorenz attractor from purely symbolic (4^3 grid categories) observations, which SVAE and multiple shooting could not achieve.
- On fMRI data, MTF with categorical task labels significantly improved D_stsp (t19=2.45, p<.013) and D_H (t19=2.72, p<.007) over BOLD-only training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MTF instead of alternative multimodal training methods | dendPLRNN · Lorenz, Rössler, and Lewis-Glass | D_stsp: Lorenz: 11.1 \pm 0.6 -> 3.4 \pm 0.35; Rössler: 3.99 \pm 1.1 -> 1.45 \pm 0.71; Lewis-Glass: 0.33 \pm 0.06 -> 0.27 \pm 0.0 | Helped |
| Add ordinal observations to heavily noise-distorted continuous Lorenz observations | MTF with dendPLRNN · Lorenz-63 | D_stsp and D_H | Helped |
| Add categorical task labels to BOLD data | MTF with dendPLRNN · fMRI | D_stsp and D_H | Helped |
| Add position data to spike-count data | MTF with dendPLRNN · CA1 neural and location data | spike train performance metrics | Helped |
| Remove the DSR reconstruction loss (ablation) | MTF · Multimodal dynamical-system reconstruction | reconstruction performance | Hurt |
| Use a non-optimal teacher-forcing interval instead of the optimal interval (ablation) | MTF · Multimodal dynamical-system reconstruction | reconstruction performance | Hurt |
| Use an ordered probit decoder instead of an ordered logit decoder (ablation) | MTF · Ordinal observations | reconstruction performance | Hurt |
| Use a temporal CNN encoder rather than mixture-of-experts, RNN, or transformer encoders (ablation) | MTF · Multimodal time series | reconstruction performance | Helped |
| Add a warm-up phase when the encoder specifies fewer latent states than the DSR model (ablation) | MTF and GVAE-TF · Prediction initialization | PE | Helped |
Surprising findings
- Dynamical systems reconstruction of chaotic attractors is possible from purely ordinal or symbolic (categorical) observations, including estimates of the maximum Lyapunov exponent.
- On the fMRI data, the GVAE-TF and BPTT-TF baselines matched or slightly beat MTF on D_stsp and D_H, although MTF had the best prediction error.
Limitations
- Forcing interval must be treated as a grid-searched hyperparameter for discrete data since the maximal Lyapunov exponent cannot be determined directly.
- Reconstruction from purely discrete data relies on a post-hoc PCA+rotation alignment, and the correlation with the co-trained linear-model measure drops to r≈0.57 for the 6d Lewis-Glass model.
Experimental setup
Datasets: Lorenz-63 simulated data Rossler simulated data Lewis-Glass 6d network simulated data fMRI BOLD and cognitive task labels (20 subjects) CA1 hippocampal spike trains and rat position
Benchmarks: Lorenz-63 Rossler Lewis-Glass fMRI cognitive task dataset CA1 hippocampus recordings
Metrics: D_stsp (KL divergence on state space occupation) D_H (Hellinger distance on power spectra) prediction error (PE) ordinal prediction error (OPE) SCC OACF CACF Lyapunov exponent estimate
Baselines: sequential multimodal VAE (SVAE) BPTT multiple shooting GVAE-TF BPTT-TF
Builds on: sparse teacher forcing generalized teacher forcing multimodal VAE sequential VAE dendPLRNN
Closest papers by meaning
- A scalable generative model for dynamical system reconstruction from neuroimaging data (NeurIPS 2024 · similarity 0.90)
- Learning Interpretable Hierarchical Dynamical Systems Models from Time Series Data (ICLR 2025 · similarity 0.89)
- Dynamical modeling of nonlinear latent factors in multiscale neural activity with real-time inference (NeurIPS 2025 · similarity 0.89)
- eXponential FAmily Dynamical Systems (XFADS): Large-scale nonlinear Gaussian state-space modeling (NeurIPS 2024 · similarity 0.88)
- True Zero-Shot Inference of Dynamical Systems Preserving Long-Term Statistics (NeurIPS 2025 · similarity 0.87)
- Multi-modal Gaussian Process Variational Autoencoders for Neural and Behavioral Data (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.