Research map / Video / Video segmentation and tracking
Deep Multimodal Representation Learning From Temporal Data
Xitong Yang, Palghat Ramesh, Radha Chitta, Sriganesh Madhvanath, Edgar A. Bernal, Jiebo Luo
Check your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CorrRNN's temporal joint representation instead of CRBM features | audio-visual speech classification · AVLetters | accuracy: $67.10$ -> $\mathbf{83.40}$ | Improved |
| Use CorrRNN's temporal joint representation instead of CRBM features | audio-visual speech classification · CUAVE | accuracy: $69.10$ -> $\mathbf{95.9}$ | Improved |
| Learn features with both video and audio, then train and test using video | CorrRNN · AVLetters cross-modality learning | accuracy: 38.08 -> 81.85 | Improved |
| Learn features with both video and audio, then train and test using video | CorrRNN · CUAVE cross-modality learning | accuracy: 42.05 -> 96.22 | Improved |
| Learn features with both video and audio, then train and test using audio | CorrRNN · AVLetters cross-modality learning | accuracy: 57.31 -> 85.33 | Improved |
| Learn features with both video and audio, then train and test using audio | CorrRNN · CUAVE cross-modality learning | accuracy: 88.32 -> 96.11 | Improved |
| Learn a shared representation from both modalities and transfer across modalities | CorrRNN · CUAVE shared representation learning | accuracy: 24.30 -> 96.77; 30.70 -> 96.33 | Improved |
| Use CorrRNN instead of MDAE + Audio RBM under 0dB SNR audio noise | CUAVE audio-visual speech classification · CUAVE noisy audio (0dB SNR) | accuracy: 82.2 -> 90.88 | Improved |
| Add cross-reconstruction loss (ablation) | CorrRNN · ISI activity classification | accuracy | Improved |
| Add self-reconstruction loss (ablation) | CorrRNN · ISI activity classification | accuracy | Improved |
| Add correlation loss (ablation) | CorrRNN · ISI activity classification | accuracy | Improved |
| Add dynamic modality weights (ablation) | CorrRNN · ISI activity classification | accuracy | Improved |
Closest papers by meaning
- Temporal Multimodal Learning in Audiovisual Speech Recognition (CVPR 2016 · similarity 0.86)
- Self-Supervised MultiModal Versatile Networks (NeurIPS 2020 · similarity 0.85)
- Mirasol3B: A Multimodal Autoregressive Model for Time-Aligned and Contextual Modalities (CVPR 2024 · similarity 0.85)
- Long-Term Recurrent Convolutional Networks for Visual Recognition and Description (TPAMI 2016 · similarity 0.85)
- Beyond Temporal Pooling: Recurrence and Temporal Convolutions for Gesture Recognition in Video (IJCV 2016 · similarity 0.85)
- Beyond Modality Fusion: Deep Ensembles for Multimodal Classification (NeurIPS 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
