Research map / Label-efficient and robust learning / Semi-supervised learning
Semi-IIN: Semi-Supervised Intra-Inter Modal Interaction Learning Network for Multimodal Sentiment Analysis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Semi-IIN, a semi-supervised multimodal sentiment analysis network with separate intra-modal and inter-modal masked attention branches fused by a gate, plus top-k confidence self-training on unlabeled AMI data.
Problem
Multimodal sentiment analysis suffers from high annotation cost and label ambiguity, and models struggle to choose the right balance between intra-modal and inter-modal interactions across samples.
Approach
Separate intra-modal masked attention (IntraMA) and inter-modal masked attention (InterMA) branches capture interactions independently, a gate dynamically weighs their outputs, and a top-k confidence self-training scheme exploits unlabeled data.
Key results
- On MOSI, Semi-IIN reaches MAE 0.679 and Corr 0.822 versus TCDN's 0.697 and 0.805 (text says 0.016 MAE improvement, which is inconsistent with the table).
- On MOSEI, Semi-IIN achieves MAE 0.497, Corr 0.804, Acc-7 55.89, surpassing TCDN (0.521 MAE, 0.782 Corr).
- Ablation on MOSI: masked attention alone raises Acc-2 from 83.97 to 85.04; combining MA and semi-supervised self-training gives the best MAE 0.497 and Corr 0.804 on MOSEI-scale ablation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Semi-IIN instead of TCDN | RoBERTa-large MSA · MOSI | MAE: 0.697 -> 0.679 | Helped |
| Use Semi-IIN instead of TCDN | RoBERTa-large MSA · MOSI | Corr: 0.805 -> 0.822 | Helped |
| Use Semi-IIN instead of TCDN | RoBERTa-large MSA · MOSEI | MAE: 0.521 -> 0.497 | Helped |
| Add masked attention without semi-supervised training (ablation) | Semi-IIN · MOSEI | MAE: 0.509 -> 0.499 | Helped |
| Add masked attention without semi-supervised training (ablation) | Semi-IIN · MOSEI | Acc-2: 83.97/86.85 -> 85.04/87.26 | Helped |
| Add semi-supervised training without masked attention (ablation) | Semi-IIN · MOSEI | MAE: 0.509 -> 0.507 | No clear effect |
| Add semi-supervised training without masked attention (ablation) | Semi-IIN · MOSEI | Acc-2: 83.97/86.85 -> 84.54/86.74 | Helped |
| Combine masked attention and semi-supervised training (ablation) | Semi-IIN · MOSEI | MAE: 0.509 -> 0.497 | Helped |
| Replace gate fusion with dot fusion (ablation) | Semi-IIN (MA only) · MOSEI | MAE: 0.499 -> 0.506 | Hurt |
| Replace gate fusion with add fusion (ablation) | Semi-IIN (MA only) · MOSEI | MAE: 0.499 -> 0.512 | Hurt |
| Replace gate fusion with concat fusion (ablation) | Semi-IIN (MA only) · MOSEI | MAE: 0.499 -> 0.506 | Hurt |
Surprising findings
- Performance decreases when the labeled ratio in semi-supervised training is 50% or 75%, which the authors attribute to imbalance in created pseudo-samples.
- Semi-supervised training alone (without MA) gives only a small change (MAE 0.507 vs 0.509 baseline), so most gains come from the masked attention design.
Limitations
- Semi-supervised gains are small and sensitive to labeled ratio; performance drops at 50% and 75% labeled ratios.
- Evaluated only on English MOSI and MOSEI; multilingual extension is left to future work.
- Some reported numbers and claims in the text are inconsistent with the table (e.g., MAE improvement over TCDN).
Experimental setup
Datasets: CMU-MOSI CMU-MOSEI AMI
Benchmarks: CMU-MOSI CMU-MOSEI
Metrics: MAE Pearson correlation Acc-2 F1 Acc-7
Baselines: LMF TFN MFM MulT ICCN MISA Self-MM MAG-BERT MMIM ConFEDE SMIN TCDN
Builds on: SMIN Self-MM MulT MAG-BERT
Closest papers by meaning
- Seek Common Ground While Reserving Differences: Semi-Supervised Image-Text Sentiment Recognition (CVPR 2025 · similarity 0.88)
- A Unified Self-Distillation Framework for Multimodal Sentiment Analysis with Uncertain Missing Modalities (AAAI 2024 · similarity 0.86)
- Multimodal Learning Without Labeled Multimodal Data: Guarantees and Applications (ICLR 2024 · similarity 0.85)
- Adaptive Graph Learning for Multimodal Conversational Emotion Detection (AAAI 2024 · similarity 0.85)
- Towards Robust Multimodal Sentiment Analysis with Incomplete Data (NeurIPS 2024 · similarity 0.84)
- MSAmba: Exploring Multimodal Sentiment Analysis with State Space Models (AAAI 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.