Research map / Label-efficient and robust learning / Semi-supervised learning
Seek Common Ground While Reserving Differences: Semi-Supervised Image-Text Sentiment Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SCRD, a semi-supervised framework for image-text sentiment recognition that decouples common/private features, trains unimodal classifiers on private features, selects a dominant modality per sample with attention, and filters pseudo-labels by unimodal agreement.
Problem
Semi-supervised image-text sentiment recognition is hurt by sentiment inconsistency between image and text, since standard multimodal alignment and SSL methods mislead the model on inconsistent pairs.
Approach
Decouple each modality's features into common and private parts; train unimodal classifiers on private features with loss-based noisy-sample selection; use a modal-selection attention module to pick the dominant sentiment modality per sample for fusion; and filter multimodal pseudo-labels by the matching degree with the dominant modality's unimodal predictions.
Key results
- SCRD reaches 60.51% on MVSA-S with 50 labels versus 60.16% for FixMatchLS, and 66.60% with 600 labels.
- On TumEmo with 700 labels, SCRD achieves 72.63% vs. 66.03% for FixMatchLS (the strongest prior SSL baseline in that table).
- Ablation on MVSA-S shows the full model reaches 67.14% vs. 64.27% for the base SSL configuration.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SCRD instead of FixMatchLS | image-text sentiment recognition · MVSA-S (50, 200, and 600 labels) | accuracy: 60.16±0.73 -> 60.51±0.32; 62.01±0.73 -> 64.27±0.72; 63.84±0.55 -> 66.60±0.59 | Helped |
| Use SCRD instead of the strongest listed baseline | image-text sentiment recognition · MVSA-M (100, 600, and 1500 labels) | accuracy: 62.42±0.52 -> 64.22±0.47; 64.80±0.58 -> 65.29±0.36; 66.28±0.38 -> 67.76±0.39 | Helped |
| Use SCRD instead of the strongest listed baseline | image-text sentiment recognition · Twitter (60, 300, and 900 labels) | accuracy: 52.48±0.38 -> 52.75±0.44; 60.07±0.36 -> 60.15±0.41; 63.48±0.49 -> 65.08±0.37 | Helped |
| Use SCRD instead of the strongest listed baseline | image-text sentiment recognition · Tourism (30, 90, and 150 labels) | accuracy: 46.06±0.66 -> 46.56±0.31; 47.83±0.39 -> 48.09±0.37; 48.02±0.62 -> 53.45±0.26 | Helped |
| Use SCRD instead of the strongest listed baseline | image-text sentiment recognition · TumEmo (700, 7000, and 14000 labels) | accuracy: 66.03±0.44 -> 72.63±0.35; 74.21±0.63 -> 76.76±0.27; 76.38±0.54 -> 78.37±0.26 | Helped |
| Add feature decoupling loss Ldec (ablation) | SCRD · MVSA-S and MVSA-M | accuracy | Helped |
| Add image unimodal classification loss Lcls_v (ablation) | SCRD · MVSA-S and MVSA-M | accuracy | Helped |
| Add text unimodal classification loss Lcls_t (ablation) | SCRD · MVSA-S and MVSA-M | accuracy | Helped |
| Add both image and text unimodal classification losses (ablation) | SCRD · MVSA-S and MVSA-M | accuracy | Helped |
| Add MSeA without the other supporting components (ablation) | SCRD · MVSA-S and MVSA-M | accuracy | Mixed |
| Add pseudo-label filtering (PLF) (ablation) | SCRD with MSeA · MVSA-S, MVSA-M, and Twitter | pseudo-label correctness: 70.18 -> 72.92; 67.19 -> 73.77; 66.30 -> 70.06 | Helped |
| Replace MSeA with element-wise product fusion (ablation) | SCRD fusion · MVSA-S | accuracy: 65.71 -> 59.55 | Hurt |
Surprising findings
- Element-wise product fusion performs notably worse than concatenation or sum (e.g., 59.55% vs 65.71% on MVSA-S), suggesting direct multiplication fails to capture sentiment relationships.
- Using MSeA alone gives only limited improvement; its benefit depends on feature decoupling and unimodal classifiers.
Limitations
- Relabeled MVSA keeps only consistently annotated conflict samples, and sentiment ambiguity and annotation quality cause failure cases.
- Experiments use a fixed backbone (ResNet and BERT) and are limited to sentiment datasets, not broader multimodal tasks.
Experimental setup
Datasets: MVSA-Single MVSA-Multiple Twitter-15 Twitter-17 Tourism TumEmo FI IMDB
Benchmarks: MVSA-S MVSA-M Twitter Tourism TumEmo
Metrics: accuracy variance over 5-fold experiments pseudo-label correctness rate
Baselines: MGNNS Hgraph-CL INIT DPRN CLMLF MVCN CMGCN DMD S2-VER CHMatch MixText SAT
Builds on: FixMatch FixMatchLS decoupling from Li et al. (DMD) mixture-of-experts
Closest papers by meaning
- Semi-IIN: Semi-Supervised Intra-Inter Modal Interaction Learning Network for Multimodal Sentiment Analysis (AAAI 2025 · similarity 0.88)
- Semi-Supervised Multimodal Classification Through Learning from Modal and Strategic Complementarities (AAAI 2025 · similarity 0.86)
- STiL: Semi-supervised Tabular-Image Learning for Comprehensive Task-Relevant Information Exploration in Multimodal Classification (CVPR 2025 · similarity 0.86)
- SemMSA: Latent Semantic-Aided Robust Multimodal Sentiment Analysis with Incomplete Data (NeurIPS 2026 · similarity 0.86)
- Toward Robust Incomplete Multimodal Sentiment Analysis via Hierarchical Representation Learning (NeurIPS 2024 · similarity 0.86)
- DLF: Disentangled-Language-Focused Multimodal Sentiment Analysis (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.