Research map / Vision-language and multimodal / Cross-modal learning

SemMSA: Latent Semantic-Aided Robust Multimodal Sentiment Analysis with Incomplete Data

NeurIPS 2026 · Vision-language and multimodal › Cross-modal learning · approach: Multimodal sentiment analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SemMSA, which uses a frozen Qwen3-1.7B LLM with visual/acoustic adapters to produce latent semantic states via iterative hidden-state refinement, and aligns all modality representations with an anchor-free kernel Gram spectral loss plus a separation regularizer, evaluated on MOSI, MOSEI, and SIMS under missing-modality settings.

Problem

Multimodal sentiment analysis when language, visual, or acoustic inputs are partially missing or corrupted (intra-modal missingness), where reconstruction and fusion methods produce spurious or noisy guidance.

Approach

Enrich incomplete multimodal prefixes with LLM hidden-state latent semantics (no text decoding) and align all four representations simultaneously by boosting the dominant eigenvalue of an RBF kernel Gram matrix, with an instance-level eigenvector separation loss to prevent collapse.

Key results

• On MOSI intra-modal missingness (averaged over missing rates 0.0-0.9), SemMSA reaches 74.36/73.91 Acc-2 and MAE 1.011, outperforming TF-Mamba and P-RMF.
• On MOSEI inter-modal missingness, SemMSA averages 77.89 F1, outperforming second-best CorrKD (75.18).
• Ablation on MOSI: adding CSR raises Acc-2 from 67.49 to 73.36 (first number, averaged protocol); full model with L_sep reaches 74.36/73.91.

Surprising findings

• Direct LLM prediction (Qwen2.5-Omni-7B) performs worse than using the LLM only as a semantic generator feeding the MSA head.
• Linear and MLP projections of modality features into LLM space perform clearly worse than adapters.

Limitations

• Gains under severe missingness rely on frozen encoders and an LLM whose semantics are not directly grounded in sentiment supervision.
• Hyperparameters such as sigma and tau are fixed at defaults with limited sensitivity analysis shown in main text.

Experimental setup

Datasets: MOSI MOSEI SIMS

Benchmarks: MOSI MOSEI SIMS

Metrics: Acc-2 F1 Acc-5 Acc-7 Acc-3 MAE Corr

Baselines: MISA Self-MM MMIM TETFN TFR-Net ALMT LNLN P-RMF TF-Mamba CubeMLP DMD MCTN

Builds on: MISA LNLN P-RMF CLIP

Closest papers by meaning

• Toward Robust Incomplete Multimodal Sentiment Analysis via Hierarchical Representation Learning (NeurIPS 2024 · similarity 0.90)
• A Unified Self-Distillation Framework for Multimodal Sentiment Analysis with Uncertain Missing Modalities (AAAI 2024 · similarity 0.90)
• Towards Robust Multimodal Sentiment Analysis with Incomplete Data (NeurIPS 2024 · similarity 0.88)
• Correlation-Decoupled Knowledge Distillation for Multimodal Sentiment Analysis with Incomplete Modalities (CVPR 2024 · similarity 0.88)
• DLF: Disentangled-Language-Focused Multimodal Sentiment Analysis (AAAI 2025 · similarity 0.88)
• A Multi-Focus-Driven Multi-Branch Network for Robust Multimodal Sentiment Analysis (AAAI 2025 · similarity 0.87)