Research map / Video / Video-language understanding
Learning Relationships between Text, Audio, and Video via Deep Canonical Correlation for Multimodal Language Analysis
Zhongkai Sun, Prathusha Sarma, William Sethares, Yingyu Liang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add text-based audio and text-based video outer-product interactions, rather than applying DCCA directly to audio and video (ablation) | ICCN · CMU-MOSI | Acc-2: 82.13 -> 83.07 | Improved |
| Add text-based audio and text-based video outer-product interactions, rather than applying DCCA directly to audio and video (ablation) | ICCN · CMU-MOSEI | Acc-2: 83.01 -> 84.18 | Improved |
| Use CCA loss instead of cosine-similarity loss (ablation) | ICCN · CMU-MOSI | Acc-2: 82.32 -> 83.07 | Improved |
| Use CCA loss instead of cosine-similarity loss (ablation) | ICCN · CMU-MOSEI | Acc-2: 82.98 -> 84.18 | Improved |
| Add text-based outer products and use CCA loss instead of direct audio-video inputs with cosine-similarity loss (ablation) | ICCN · CMU-MOSI | Acc-2: 81.49 -> 83.07 | Improved |
| Add text-based outer products and use CCA loss instead of direct audio-video inputs with cosine-similarity loss (ablation) | ICCN · CMU-MOSEI | Acc-2: 82.59 -> 84.18 | Improved |
| Use ICCN rather than the strongest reported baseline | CMU-MOSI · CMU-MOSI | Acc-2: 82.53 -> 83.07 | Improved |
| Use ICCN rather than the strongest reported baseline | CMU-MOSEI · CMU-MOSEI | Acc-2: 84.40 -> 84.18 | Hurt |
| Use ICCN rather than the strongest reported baseline | IEMOCAP Happy · IEMOCAP Happy | Acc-2: 86.67 -> 87.41 | Improved |
| Use ICCN rather than the strongest reported baseline | IEMOCAP Angry · IEMOCAP Angry | Acc-2: 86.99 -> 88.62 | Improved |
| Use ICCN rather than the strongest reported baseline | IEMOCAP Neutral · IEMOCAP Neutral | Acc-2: 70.26 -> 69.73 | Hurt |
Closest papers by meaning
- DLF: Disentangled-Language-Focused Multimodal Sentiment Analysis (AAAI 2025 · similarity 0.88)
- Debiased Multimodal Understanding for Human Language Sequences (AAAI 2025 · similarity 0.88)
- Towards Multimodal Sentiment Analysis via Hierarchical Correlation Modeling with Semantic Distribution Constraints (AAAI 2025 · similarity 0.88)
- Enriching Multimodal Sentiment Analysis Through Textual Emotional Descriptions of Visual-Audio Content (AAAI 2025 · similarity 0.87)
- MSE-Adapter: A Lightweight Plugin Endowing LLMs with the Capability to Perform Multimodal Sentiment Analysis and Emotion Recognition (AAAI 2025 · similarity 0.87)
- AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
