Research map / Large language models / Language, safety and interpretability
MSE-Adapter: A Lightweight Plugin Endowing LLMs with the Capability to Perform Multimodal Sentiment Analysis and Emotion Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MSE-Adapter, a lightweight plugin (about 2.6M to 2.8M trainable parameters for 6/7B LLMs) that lets frozen LLMs perform multimodal sentiment analysis and emotion recognition via a Text-Guide-Mixer (Hadamard product) and Multi-Scale Fusion modules.
Problem
Fine-tuning pre-trained language models for multimodal sentiment analysis (MSA) and emotion recognition in conversations (ERC) is computationally expensive and causes the models to lose their general capabilities.
Approach
A frozen LLM is paired with a small adapter that encodes audio and visual sequences with LSTMs, fuses them with the text representation via element-wise Hadamard products (Text-Guide-Mixer), applies multi-scale MLP fusion, and projects the result into pseudo tokens prepended to the LLM input; the LLM generates sentiment labels autoregressively.
Key results
- MSE-ChatGLM3-6B reports 86.91 Acc-2 on MOSEI and 83.77 Acc-2 on SIMS-V2, outperforming the listed baselines on most metrics.
- MSE-ChatGLM3-6B reaches 72.73 WF1 on CHERMA, 2.19% above the best baseline LFMIM (70.54).
- Removing the MSF module lowers MELD WF1 from 65.13 to 61.75 in the ablation on MSE-ChatGLM3-6B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MSE-ChatGLM3-6B instead of the strongest listed baseline UniMSE | MOSEI MSA · MOSEI | Acc-2: 85.86 -> 86.91 | Helped |
| Use MSE-ChatGLM3-6B instead of AV-MC | SIMS-V2 MSA · SIMS-V2 | Acc-2: 82.50 -> 83.77 | Helped |
| Use MSE-ChatGLM3-6B instead of UniMSE | MELD ERC · MELD | Acc: 65.09 -> 66.23 | Helped |
| Use MSE-ChatGLM3-6B instead of LFMIM | CHERMA ERC · CHERMA | WF1: 70.54 -> 72.73 | Helped |
| Add audio modality (ablation) | MSE-Adapter on MELD · MELD | Acc: 66.24 -> 66.23 | No clear effect |
| Add vision modality (ablation) | MSE-Adapter on MELD · MELD | Acc: 66.13 -> 66.23 | Helped |
| Add text modality (ablation) | MSE-Adapter on MELD · MELD | Acc: 46.44 -> 66.23 | Helped |
| Add audio and vision modalities (ablation) | MSE-Adapter on MELD · MELD | Acc: 57.11 -> 66.23 | Helped |
| Add TGM (ablation) | MSE-Adapter on MELD · MELD | Acc: 65.07 -> 66.23 | Helped |
| Add MSF (ablation) | MSE-Adapter on MELD · MELD | Acc: 62.72 -> 66.23 | Helped |
| Add audio modality (ablation) | MSE-Adapter on SIMS-V2 · SIMS-V2 | Acc-2: 83.46 -> 83.77 | Helped |
| Add vision modality (ablation) | MSE-Adapter on SIMS-V2 · SIMS-V2 | Acc-2: 78.84 -> 83.77 | Helped |
Surprising findings
- MSE-LLaMA2-7B, despite being larger, underperforms MSE-ChatGLM3-6B on the Chinese SIMS-V2 dataset, which the authors attribute to LLaMA2's weaker Chinese ability.
- Removing the MSF module hurts performance more than removing the TGM module.
Limitations
- Performance depends heavily on the inherent capabilities of the frozen LLM, e.g. weaker results for LLaMA2-7B on Chinese data.
- Qwen-1.8B and LLaMA2-7B show performance drops when training data falls below about 40% of the training set, per the appendix analysis.
- Comparisons mix reported numbers from papers and official repositories, and some baseline values are cited rather than reproduced.
Experimental setup
Datasets: MOSEI SIMS-V2 MELD CHERMA
Benchmarks: MOSEI SIMS-V2 MELD CHERMA
Metrics: Acc-2 F1 Acc-7 MAE Corr Acc2_weak Acc WF1
Baselines: TFN LMF MISA MAG-BERT Self-MM MMIM CHFN UniMSE UniSA AV-MC MMGCN MM-DFN
Builds on: UniMSE UniSA MAG CHFN
Closest papers by meaning
- MSAmba: Exploring Multimodal Sentiment Analysis with State Space Models (AAAI 2025 · similarity 0.88)
- DLF: Disentangled-Language-Focused Multimodal Sentiment Analysis (AAAI 2025 · similarity 0.87)
- AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models (ICML 2025 · similarity 0.87)
- MME-Emotion: A Holistic Evaluation Benchmark for Emotional Intelligence in Multimodal Large Language Models (ICLR 2026 · similarity 0.87)
- Bridging the Gap for Test-Time Multimodal Sentiment Analysis (AAAI 2025 · similarity 0.87)
- Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.