Research map / Video / Video-language understanding
Enhancing Multimodal Affective Analysis with Learned Live Comment Features
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds LCAffect, a bilingual live-comment dataset (~11M comments, 829 h), trains a contrastive video-to-live-comment encoder to produce synthetic live comment features, and adds them to a multimodal fusion model for sentiment, emotion, and sarcasm tasks.
Problem
Live comments (Danmaku) are informative for affective video analysis but are unavailable on many platforms and videos, and existing live-comment datasets are mostly Chinese, user-generated everyday videos with limited emotional range.
Approach
Pre-train a video encoder with CLIP-style contrastive learning to map video segments (transcript + frames) into a live-comment embedding space, using a multi-label contrastive objective, then use its outputs as synthetic live-comment features that augment a multimodal (text, acoustic, visual) fusion model for downstream affective tasks even when no comments exist.
Key results
- On CH-SIMS v2, adding live comment features (+LC) raises accuracy by 3.18 points over the vanilla fusion model (90.12 vs 86.94 Acc2).
- On MELD, +LC improves weighted F1 by 2.89 points (69.62 vs 66.73).
- On MUStARD, +LC improves weighted F1 by 3.0 points (70.5 vs 67.5).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add synthetic live comment features | Multimodal Fusion Encoder · CH-SIMS v2 | Acc2: 3.18-point increase | Helped |
| Add synthetic live comment features | Multimodal Fusion Encoder · CH-SIMS | Acc2: 83.16 -> 84.88 | Helped |
| Add synthetic live comment features | Multimodal Fusion Encoder · MOSI | Acc2: 88.26 -> 89.02 | Helped |
| Add synthetic live comment features | Multimodal Fusion Encoder · MOSEI | Acc2: 86.24 -> 87.15 | Helped |
| Add synthetic live comment features | Multimodal Fusion Encoder · MELD | weighted F1: 2.89-point increase | Helped |
| Add synthetic live comment features | Multimodal Fusion Encoder · MUStARD | weighted F1: 3.0-point increase | Helped |
Surprising findings
- Synthetic live comment features are less discriminative than text sentiment features in a logistic regression probe, yet give the largest gains when used with the multimodal fusion encoder.
- English sentiment gains on MOSI and MOSEI are modest, attributed to genre mismatch between the pre-training corpus (conversational) and monologue benchmarks.
Limitations
- English sentiment gains on MOSI/MOSEI are modest, likely due to genre shift between pre-training data and monologue benchmarks.
- Pre-training corpus is dominated by Chinese user-generated content; English coverage is limited to TV shows and movies with subtitles.
Experimental setup
Datasets: LCAffect CH-SIMS CH-SIMS v2 MOSI MOSEI MELD MUStARD
Benchmarks: CH-SIMS CH-SIMS v2 MOSI MOSEI MELD MUStARD
Metrics: Acc2 Acc7 F1 Corr MAE R2 weighted F1 weighted precision weighted recall
Baselines: MulT SVM AV-MC UniMSE VLP2MSA MMML Erlangshen-Roberta-sentiment COLDETECTOR ViT-face-expression
Builds on: CLIP MulT TimeSformer
Closest papers by meaning
- AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models (ICML 2025 · similarity 0.86)
- Infer Induced Sentiment of Comment Response to Video: A New Task, Dataset and Baseline (NeurIPS 2024 · similarity 0.86)
- MART: Masked Affective RepresenTation Learning via Masked Temporal Distribution Distillation (CVPR 2024 · similarity 0.84)
- Enriching Multimodal Sentiment Analysis Through Textual Emotional Descriptions of Visual-Audio Content (AAAI 2025 · similarity 0.84)
- $E^3$: Exploring Embodied Emotion Through A Large-Scale Egocentric Video Dataset (NeurIPS 2024 · similarity 0.84)
- Customizing Visual Emotion Evaluation for MLLMs: An Open-vocabulary, Multifaceted, and Scalable Approach (ICLR 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.