Research map / Vision-language and multimodal / CLIP and zero-shot
An Empirical Study on Configuring In-Context Learning Demonstrations for Unleashing MLLMs' Sentimental Perception Capability
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies how to configure in-context learning demonstrations (retrieval similarity, modality presentation, sentiment distribution) so MLLMs perform multimodal sentiment analysis, reporting large gains over zero-shot and random ICL on six MSA datasets.
Problem
Zero-shot MLLMs perform poorly on multimodal sentiment analysis (MSA) relative to supervised models, and how to configure ICL demonstrations for MSA is underexplored.
Approach
Extends ICL to MSA and devises six multimodal similarity measures (adding aspect similarity and weighted image/text/aspect combinations), tests modality compositions including MLLM-generated captions and diffusion-generated images, and five sentiment distribution protocols; identifies a negative-sentiment predictive bias in MLLMs and counteracts it via category-balanced demonstrations.
Key results
- Optimized ICL averages 15.9% accuracy gain over zero-shot and 11.2% over random ICL across six MSA datasets (IDEFICS, full training set support: mean 70.9 vs. 55.0 zero-shot).
- With full training set as support, the ICL strategy with IDEFICS reaches a mean of 70.9 across six datasets, exceeding SOTA few-shot models (68.3) but still below fully-supervised models (78.8).
- Category Balanced distribution on Twitter-15/17 counteracts MLLM bias against negative predictions, while Unlimited is preferred for MVSA and MASAD.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Configure retrieval, modality presentation, and sentiment distribution with the proposed strategies | IDEFICS · Six MSA datasets, mean accuracy | accuracy: +15.9% vs zero-shot; +11.2% vs random ICL | Helped |
| Use optimized demonstration configuration instead of zero-shot | IDEFICS · MVSA-S, MVSA-M, TumEmo, Twitter-15, Twitter-17, MASAD | accuracy: 38.6 -> 66.5; 56.5 -> 67.7; 46.6 -> 63.4; 60.7 -> 67.0; 54.7 -> 62.0; 73.0 -> 88.2 | Helped |
| Prioritize textual similarity in WIT retrieval with weights 2:8 (ablation) | ICL similarity retrieval · MVSA-S and MVSA-M | accuracy: WIT 2:8 reaches 66.5 and 67.7 | Helped |
| Prioritize aspect similarity in WITA retrieval (image:text 7:3; image+text:aspect 2:8) (ablation) | ICL similarity retrieval · Twitter-15 and Twitter-17 | accuracy: 64.4; 58.7 | Helped |
| Add aspect similarity to image/text similarity for retrieval (ablation) | Aspect-level ICL · Twitter-15 and Twitter-17 | accuracy | Helped |
| Replace original image and text with captioned text and generated image (ablation) | Multimodal ICL · MVSA-S, MVSA-M, Twitter-15, Twitter-17 | accuracy: 55.7 -> 47.5; 66.2 -> 53.7; 61.4 -> 53.1; 57.6 -> 53.1 | Hurt |
| Add captioned text to original image and text (ablation) | Multimodal ICL · MVSA-S, MVSA-M, Twitter-15, Twitter-17 | accuracy: 55.7 -> 55.3; 66.2 -> 64.4; 61.4 -> 60.3; 57.6 -> 55.6 | Hurt |
| Add generated image to original image and text (ablation) | Multimodal ICL · MVSA-S, MVSA-M, Twitter-15, Twitter-17 | accuracy: 55.7 -> 50.8; 66.2 -> 62.0; 61.4 -> 59.7; 57.6 -> 54.8 | Hurt |
| Use original image and text rather than adding both auxiliary modalities (ablation) | Multimodal ICL · MVSA-S, MVSA-M, Twitter-15, Twitter-17 | accuracy: 49.2 -> 55.7; 61.4 -> 66.2; 58.7 -> 61.4; 53.1 -> 57.6 | Helped |
| Use unlimited sentiment distribution rather than category-balanced demonstrations (ablation) | IDEFICS sentiment distribution · MVSA-S | accuracy | Helped |
| Use category-balanced demonstrations to counter the negative-prediction bias (ablation) | IDEFICS sentiment distribution · Twitter-17 | accuracy | Helped |
| Use random rather than optimized retrieval, presentation, and distribution | IDEFICS ICL · Six MSA datasets, mean accuracy | accuracy: 59.7 -> 69.1 | Hurt |
Surprising findings
- Adding auxiliary modalities (captions or generated images) to original image-text input generally degrades ICL performance, attributed to impaired task learning.
- Weighting similarity toward text (WIT) or the IT/ITA unweighted combinations do not always yield the best retrieval; balanced weights matter more than including all components.
Limitations
- Only two MLLMs (IDEFICS-9B and Open-Flamingo2-9B) investigated due to practical constraints; advanced MLLMs not covered.
- A performance gap to fully-supervised models remains, especially on some datasets.
- Source of the identified sentiment predictive bias is not investigated.
Experimental setup
Datasets: MVSA-S MVSA-M TumEmo Twitter-15 Twitter-17 MASAD
Benchmarks: MVSA-S MVSA-M TumEmo Twitter-15 Twitter-17 MASAD
Metrics: accuracy precision recall Same Label Rate
Baselines: Zero-shot paradigm ICL Random ICL RICES ICL SQPA ICL MMICES SOTA few-shot models SOTA fully-supervised models
Builds on: RICES MMICES SQPA
Closest papers by meaning
- Customizing Visual Emotion Evaluation for MLLMs: An Open-vocabulary, Multifaceted, and Scalable Approach (ICLR 2026 · similarity 0.89)
- AIM: Let Any Multimodal Large Language Models Embrace Efficient In-Context Learning (AAAI 2025 · similarity 0.89)
- SymDPO: Boosting In-Context Learning of Large Multimodal Models with Symbol Demonstration Direct Preference Optimization (CVPR 2025 · similarity 0.87)
- MME-Emotion: A Holistic Evaluation Benchmark for Emotional Intelligence in Multimodal Large Language Models (ICLR 2026 · similarity 0.87)
- II-Bench: An Image Implication Understanding Benchmark for Multimodal Large Language Models (NeurIPS 2024 · similarity 0.87)
- Beyond task performance: evaluating and reducing the flaws of large multimodal models with in-context-learning (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.