Research map / Vision-language and multimodal / Cross-modal learning
Uncertain Multimodal Intention and Emotion Understanding in the Wild
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the MINE dataset of 20k+ social media posts with natural modality gaps and joint intention/emotion labels, and proposes BEAR (BEIFormer + Modality Asynchronous Prompt) as a baseline.
Problem
Recognizing intention and emotion from social media posts where modalities (text, image, video, audio) are incomplete or missing, and where the relationship between intention and emotion is underexploited.
Approach
Uses masked-label contrastive alignment (BEIFormer) to implicitly mine emotion-intention correlations during training, and a Modality Asynchronous Prompt that adds prompts only to missing modalities via element-wise addition instead of concatenating prompts for all modalities.
Key results
- BEAR reaches 35.79% Macro F1 for intention on the MINE test split, +2.68% over the MulT baseline (33.11%... reported as 34.11% MulT).
- BEAR reaches 64.53% emotion accuracy on the MINE test split, +1.81% over MulT.
- MAP improves MulT on CMU-MOSEI across modality combinations, e.g., +2.85% for MFN with text-visual inputs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use BEAR instead of MulT | MINE intention recognition · MINE validation | Macro F1: 34.37 -> 36.02 | Helped |
| Use BEAR instead of MFSA | MINE emotion recognition · MINE validation | accuracy: 64.15 -> 64.82 | Helped |
| Use BEAR instead of TCL MAP | MINE intention recognition · MINE testing | Macro F1: 34.73 -> 35.25 | Helped |
| Use BEAR instead of MFSA | MINE emotion recognition · MINE testing | accuracy: 64.05 -> 64.67 | Helped |
| Add MAP to the baseline (ablation) | MINE intention recognition · MINE testing | Macro F1: 32.81 -> 33.36 | Helped |
| Add BEIF to the baseline (ablation) | MINE intention recognition · MINE testing | Macro F1: 32.81 -> 35.25 | Helped |
| Add MAP and BEIF together (ablation) | MINE intention recognition · MINE testing | Macro F1: 32.81 -> 35.79 | Helped |
| Use multi-task learning instead of BEIFormer (ablation) | MINE intention recognition · MINE testing | Micro F1: 52.13 -> 51.07 | Hurt |
| Replace BEIFormer with EmoInt-Trans (ablation) | MINE intention recognition · MINE testing | Macro F1: 32.81 -> 31.59 | Hurt |
| Add BEIFormer (ablation) | MINE intention recognition · MINE testing | Macro F1: 32.81 -> 35.25 | Helped |
| Use supplementary information at inference (ablation) | MINE intention recognition · MINE testing | Macro F1: 32.81 -> 41.87 | Helped |
| Replace visual feature extractor with CLIP-ViT-S (ablation) | BEAR · MINE testing | Macro F1: +1.81% Macro F1 | Helped |
Surprising findings
- Generic multi-task learning of emotion and intention reduces Micro F1 (-1.06%) and Samples F1 (-0.30%) relative to the baseline.
- Random modality sampling used in prior missing-modality work is argued to introduce bias that alters content meaning.
Limitations
- Visual content is about 70% of the data and audio is a small portion, so the modality distribution is imbalanced.
- Emotion-intention correlation mining gains are modest (+2.44% Macro F1) relative to using complete inference-time priors.
Experimental setup
Datasets: MINE CMU-MOSEI
Benchmarks: MINE CMU-MOSEI
Metrics: Macro F1 Micro F1 Samples F1 Accuracy F1 Score Precision Recall
Baselines: MISA MFSA MulT TCL MAP MMIN MPMM EmoInt-Trans
Builds on: MulT MPMM MMIN MIntRec
Closest papers by meaning
- AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models (ICML 2025 · similarity 0.89)
- Emotion-Coherent Reasoning for Multimodal LLMs via Emotional Rationale Verifier (AAAI 2026 · similarity 0.87)
- CAMEL: Capturing Metaphorical Alignment with Context Disentangling for Multimodal Emotion Recognition (AAAI 2024 · similarity 0.87)
- Customizing Visual Emotion Evaluation for MLLMs: An Open-vocabulary, Multifaceted, and Scalable Approach (ICLR 2026 · similarity 0.87)
- Enriching Multimodal Sentiment Analysis Through Textual Emotional Descriptions of Visual-Audio Content (AAAI 2025 · similarity 0.86)
- Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.