Research map / Label-efficient and robust learning / Test-time adaptation
Bridging the Gap for Test-Time Multimodal Sentiment Analysis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CASP, a test-time adaptation method for multimodal sentiment regression combining contrastive adaptation via modality random dropout with self-training on stable pseudo-labels selected across training checkpoints.
Problem
Test-time adaptation for multimodal sentiment analysis, which is a multimodal regression task, under distribution shifts between source and target datasets without access to source data or target labels; existing TTA methods rely on probabilistic/classification objectives and unimodal augmentations.
Approach
Adapts a pretrained MSA regression model with a contrastive NT-Xent loss between original and modality-dropped inputs (updating only normalization layers), then selects pseudo-labels whose predictions are stable across checkpoints (low average checkpoint-to-checkpoint change) and self-trains on them.
Key results
- On MOSEI→SIMS with early fusion, CASP raises accuracy from 45.95 to 63.89, versus about 3% gain for the second-best method ST.
- On MOSI→SIMS with late fusion, CASP reaches 51.27 ACC vs 39.17 source, with GC second at 42.23.
- Ablation shows contrastive adaptation alone gives up to ~10-15 accuracy points, and stable pseudo-label self-training adds about 1-7 points; pseudo-label MAE drops around 40-60% relative to source-model labels.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply CASP instead of the strongest baseline | Late Fusion · all five reported domain shifts | accuracy | Helped |
| Apply CASP instead of the strongest baseline | Early Fusion · all five reported domain shifts | accuracy | Helped |
| Add contrastive adaptation (ablation) | CASP, Late Fusion · all five domain shifts | accuracy | Helped |
| Add contrastive adaptation (ablation) | CASP, Early Fusion · all five domain shifts | accuracy | Helped |
| Add stable pseudo-label generation (ablation) | CASP, Late Fusion · all five domain shifts | accuracy | Helped |
| Add stable pseudo-label generation (ablation) | CASP, Early Fusion · all five domain shifts | accuracy | Helped |
| Drop 2 modalities rather than 1 (ablation) | CASP, Late Fusion · all five domain shifts | accuracy | Hurt |
| Drop 2 modalities rather than 1 (ablation) | CASP, Early Fusion · all five domain shifts | accuracy | Hurt |
| Increase pseudo-label checkpoint interval M (ablation) | Stable pseudo-label generation, both backbones · five domain shifts | MAE decline rate | Helped |
| Increase stability threshold λ (ablation) | Stable pseudo-label generation, both backbones · five domain shifts | MAE decline rate | Helped |
Surprising findings
- Baselines ST, Norm, GC and RF degrade performance below the source model on some metrics in several shift settings.
- Dropping two modalities performs worse than dropping one, since strong consistency enforcement loses information on three-modality data.
Limitations
- Requires a multimodal dataset with modalities that can be dropped; performance trade-off with number of dropped modalities.
- Experiments limited to three MSA datasets and two backbones, all on extracted features.
- Hyperparameters (M, lambda, epochs) are tuned and sensitive; the stability threshold requires choosing lambda.
Experimental setup
Datasets: CMU-MOSI CMU-MOSEI CH-SIMS
Benchmarks: CMU-MOSI CMU-MOSEI CH-SIMS
Metrics: ACC F1 MAE
Baselines: Source ST Norm GC (TTA-IQA group contrastive) RF (READ reliable fusion)
Builds on: TENT SHOT NT-Xent READ TTA-IQA
Closest papers by meaning
- Smoothing the Shift: Towards Stable Test-Time Adaptation under Complex Multimodal Noises (ICLR 2025 · similarity 0.87)
- MSE-Adapter: A Lightweight Plugin Endowing LLMs with the Capability to Perform Multimodal Sentiment Analysis and Emotion Recognition (AAAI 2025 · similarity 0.87)
- Toward Robust Incomplete Multimodal Sentiment Analysis via Hierarchical Representation Learning (NeurIPS 2024 · similarity 0.85)
- Seek Common Ground While Reserving Differences: Semi-Supervised Image-Text Sentiment Recognition (CVPR 2025 · similarity 0.85)
- MSAmba: Exploring Multimodal Sentiment Analysis with State Space Models (AAAI 2025 · similarity 0.85)
- DLF: Disentangled-Language-Focused Multimodal Sentiment Analysis (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.