Research map / Vision-language and multimodal / Cross-modal learning
Temporal Multimodal Learning in Audiovisual Speech Recognition
Check your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Model temporal audiovisual sequences by connecting a sequence of MRBMs through joint representations | RTMRBM · AVLetters | mean accuracy: 64.8 -> 66.04 | Helped |
| Model temporal audiovisual sequences by connecting a sequence of MRBMs through joint representations | RTMRBM · AVLetters2 | accuracy: 49.61 -> 56.66; 67.89 -> 74.77 | Helped |
| Model temporal audiovisual sequences by connecting a sequence of MRBMs through joint representations | RTMRBM · AVDigits | accuracy: 51.57 -> 55.36; 66.74 -> 71.77 | Helped |
| Use audiovisual input instead of visual-only input | RTMRBM · AVLetters | mean accuracy: 64.63 -> 66.04 | Helped |
| Use audiovisual input instead of audio-only input | RTMRBM · AVLetters | mean accuracy: 64.41 -> 66.04 | Helped |
| Use audiovisual input instead of audio-only input | RTMRBM · AVLetters2, clean | accuracy: 42.25 -> 56.66 | Helped |
| Use audiovisual input instead of audio-only input | RTMRBM · AVLetters2, 12dB | accuracy: 75.85 -> 74.77 | Hurt |
| Use audiovisual input instead of audio-only input | RTMRBM · AVDigits, clean | accuracy: 43.50 -> 55.36 | Helped |
| Use audiovisual input instead of audio-only input | RTMRBM · AVDigits, 12dB | accuracy: 71.02 -> 71.77 | Helped |
| Use recurrent temporal modeling rather than a non-temporal multimodal network | RTMRBM · AVDigits utterance latter part | mean accuracy | Helped |
| Rely on recurrent joint-state connections over long sequences | RTMRBM · AVDigits and AVLetters2 utterance parts | mean accuracy | Mixed |
Closest papers by meaning
- Mirasol3B: A Multimodal Autoregressive Model for Time-Aligned and Contextual Modalities (CVPR 2024 · similarity 0.84)
- RTFS-Net: Recurrent Time-Frequency Modelling for Efficient Audio-Visual Speech Separation (ICLR 2024 · similarity 0.83)
- Massive Sound Embedding Benchmark (MSEB) (NeurIPS 2025 · similarity 0.82)
- TSAM: Temporal SAM Augmented with Multimodal Prompts for Referring Audio-Visual Segmentation (CVPR 2025 · similarity 0.82)
- FAVE: A Structured Benchmark for Fine-Grained Audio-Visual Temporal Evaluation in Multimodal LLMs (CVPR 2026 · similarity 0.82)
- Multichannel AV-wav2vec2: A Framework for Learning Multichannel Multi-Modal Speech Representation (AAAI 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.