Research map / Vision-language and multimodal / Cross-modal learning
Multimodal sentiment analysis
Uses distillation and consistency learning for sentiment recognition, including with missing modalities.
106 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in Cross-modal learning, keyword cluster “fusion · missing · multi modal”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
18 papers in 2023–24 and 73 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 6.2% to 9.3%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · Co-Regularized PLSA for Multi-Modal Learning (AAAI 2016 · 2 citations)
- 2018 · Multimodal Poisson Gamma Belief Network (AAAI 2018 · 11 citations)
- 2019 · An Efficient Approach to Informative Feature Extraction from Multimodal Data (AAAI 2019 · 64 citations)
- 2020 · Iterative Answer Prediction With Pointer-Augmented Multimodal Transformers for TextVQA (CVPR 2020 · 222 citations)
- 2021 · Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysis (AAAI 2021 · 751 citations)
- 2022 · Tailor Versatile Multi-Modal Learning for Multi-Label Emotion Recognition (AAAI 2022 · 75 citations)
- 2023 · Decoupled Multimodal Distilling for Emotion Recognition (CVPR 2023 · 220 citations)
- 2024 · A Unified Self-Distillation Framework for Multimodal Sentiment Analysis with Uncertain Missing Modalities (AAAI 2024 · 59 citations)
- 2025 · DLF: Disentangled-Language-Focused Multimodal Sentiment Analysis (AAAI 2025 · 77 citations)
- 2026 · Capturing Dynamic User Interests Under Modality Imbalance for Multimodal Sequential Recommendation (AAAI 2026 · 4 citations)
Earliest papers here
- Co-Regularized PLSA for Multi-Modal Learning (AAAI 2016 · 2 citations)
- Multimodal Poisson Gamma Belief Network (AAAI 2018 · 11 citations)
- Multi-Modal Multi-Task Learning for Automatic Dietary Assessment (AAAI 2018 · 8 citations)
What has been tried
56 papers here report 621 measured changes (a component added, removed or swapped, compared with a baseline): helped 82%, no clear effect 2%, hurt 13%, mixed 2%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Add the feature disentanglement module | CMU-MOSI · Acc-7 | 45.92 -> 47.08 | Helped | DLF: Disentangled-Language-Focused Multimodal Sentiment Analysis (AAAI 2025 · 77 citations) |
| remove feature distillation loss Lfea | MOSI Avg. · F1 | 75.56 -> 73.50 | Hurt | A Unified Self-Distillation Framework for Multimodal Sentiment Analysis with Uncertain Missing Modalities (AAAI 2024 · 59 citations) |
| Add sample-level contrastive distillation (SCD) | MOSI, inter-modality missingness (Avg.) · F1 score | 72.46 -> 74.69 | Helped | Correlation-Decoupled Knowledge Distillation for Multimodal Sentiment Analysis with Incomplete Modalities (CVPR 2024 · 55 citations) |
| add unimodal spectrum compression (USC) | Weibo · Accuracy | 0.866 -> 0.901 | Helped | Frequency Spectrum Is More Effective for Multimodal Representation and Fusion: A Multimodal Spectrum Rumor Detector (AAAI 2024 · 46 citations) |
| add Mixture of Modality Experts and Unimodal Distillation | CMU-MOSI · ACC7 | 43.0 -> 47.7 | Helped | EMOE: Modality-Specific Enhanced Dynamic Emotion Experts (CVPR 2025 · 34 citations) |
| Add DMC and reconstructor | MOSI · Acc-7 | 29.54 -> 34.26 | Helped | Towards Robust Multimodal Sentiment Analysis with Incomplete Data (NeurIPS 2024 · 29 citations) |
| Add EPE to DQPSA | Twitter2015 MATE · F1 | 86.3 -> 87.7 | Helped | A Novel Energy Based Model Mechanism for Multi-Modal Aspect-Based Sentiment Analysis (AAAI 2024 · 24 citations) |
| combine image, text, and depth modalities instead of using image modality only | Things-EEG 200-way zero-shot · Top-1 accuracy | 33.30 -> 35.64 | Helped | CognitionCapturer: Decoding Visual Stimuli from Human EEG Signal with Multimodal Information (AAAI 2025 · 19 citations) |
| add the adaptive fusion-based classification module | Weibo · accuracy | 0.938 -> 0.946 | Helped | External Reliable Information-enhanced Multimodal Contrastive Learning for Fake News Detection (AAAI 2025 · 18 citations) |
| add interaction constraints (l_ic) | Weibo17, Politifact, and GossipCop · accuracy | 0.909 -> 0.915; 0.911 -> 0.922; 0.845 -> 0.879 | Helped | RaCMC: Residual-Aware Compensation Network with Multi-Granularity Constraints for Fake News Detection (AAAI 2025 · 16 citations) |
| Add SAConv instead of SALN | Out-of-domain parallel reference speech · SMOS | 3.81 ± 0.07 -> 4.03 ± 0.08 | Helped | MM-TTS: Multi-Modal Prompt Based Style Transfer for Expressive Text-to-Speech Synthesis (AAAI 2024 · 14 citations) |
| Add macro fusion | CMU-MOSI incomplete (AUILC) · Corr | 0.451 -> 0.461 | Helped | A Multi-Focus-Driven Multi-Branch Network for Robust Multimodal Sentiment Analysis (AAAI 2025 · 11 citations) |
What reviewers will expect
Datasets and baselines named in the 34 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- CMU-MOSEI · 7 of 34 papers
- CMU-MOSI · 5 of 34 papers
- Amazon Baby · 4 of 34 papers
- Amazon Sports · 2 of 34 papers
- Amazon Clothing · 2 of 34 papers
- IEMOCAP · 2 of 34 papers
Baselines
- MISA · 6 of 34 papers
- MulT · 5 of 34 papers
- MMGCN · 5 of 34 papers
- TFN · 4 of 34 papers
- LMF · 4 of 34 papers
- Self-MM · 4 of 34 papers
- MMIM · 4 of 34 papers
- LightGCN · 4 of 34 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 91 papers; from OpenAlex and the papers' affiliations).
- Wuhan University · 5 papers
- Chinese Academy of Sciences · 3 papers
- Fudan University · 3 papers
- Shandong University · 3 papers
- Tsinghua University · 3 papers
- The Chinese University of Hong Kong · 3 papers
- Nanyang Technological University · 3 papers
- Kuaishou- 快手科技 · 3 papers
- Hebei University of Science and Technology · 2 papers
- Engineering Research Center of AI and Robotics · 2 papers
Use it
Download BibTeX of 34 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Multimodal fusion with missing modalities (208 papers)
- Multi-modality image fusion (194 papers)
- Medical multimodal fusion (166 papers)
- Multi-modal object re-identification (40 papers)
Elsewhere in the topic
- Unimodal-to-multimodal representation transfer (145 papers)
- Probabilistic embeddings for cross-modal retrieval (108 papers)
- Contrastive vision-language alignment (104 papers)
- Audio-visual speech separation (76 papers)
- Audio-visual source localization (67 papers)
- Cross-modal hashing (45 papers)
- Self-supervised audio-visual representation learning (44 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).