Research map / Vision-language and multimodal / Multimodal LLMs
VEGAS: Towards Visually Explainable and Grounded Artificial Social Intelligence
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VEGAS, a generative multimodal LLM for Social-IQ that uses language-guided frame sampling and generalist instruction fine-tuning to produce open-ended, visually grounded answers and reduce language shortcuts.
Problem
Social-IQ MCQ accuracy of current models relies heavily on language shortcuts and ignores visual context, and the closed-set format cannot reveal whether the reasoning behind a selection is correct.
Approach
Replace closed-set MCQ with open-ended generation driven by a language-guided sampler that selects question-relevant frames, a temporal attention module that restores frame order, and a social traits projector trained with emotion recognition and joint multimodal instruction tuning.
Key results
- VEGAS achieves 80.90% supervised MCQ accuracy on Social-IQ versus 79.17% for Video-LLaVA and 74.91% for MMTC-ESC.
- VEGAS-generalist reaches 54.9% open-ended accuracy in the QVS setting, above Video-LLaVA's 49.5%.
- Modality ablation shows VEGAS improves visual-context contribution by 9.28% over Video-LLaVA (-0.43%) in MCQ.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add TAM to VEGAS with n=32 frames (ablation) | VEGAS · Social-IQ-2.0 open-ended QVS | Accuracy: 49.0 -> 51.2 | Helped |
| Add TAM to VEGAS with n=32 frames (ablation) | VEGAS · Social-IQ-2.0 open-ended QVS | Score: 3.3 -> 3.5 | Helped |
| Use n=16 rather than n=32 candidate frames (ablation) | VEGAS · Social-IQ-2.0 open-ended QV | Accuracy: 43.6 -> 46.1 | Helped |
| Use n=16 rather than n=32 candidate frames (ablation) | VEGAS · Social-IQ-2.0 open-ended QV | Score: 3.2 -> 2.8 | Hurt |
| Add STP during GIFT (ablation) | VEGAS-generalist · Social-IQ-2.0 open-ended QV | Accuracy: 48.4 -> 48.5 | No clear effect |
| Add STP during GIFT (ablation) | VEGAS-generalist · Social-IQ-2.0 open-ended QVS | Accuracy: 51.5 -> 54.9 | Helped |
| Add STP during GIFT (ablation) | VEGAS-generalist · Social-IQ-2.0 open-ended QVS | Score: 3.6 -> 3.9 | Helped |
| Use full-parameter fine-tuning instead of LoRA (ablation) | VEGAS · Social-IQ-2.0 supervised MCQ | Accuracy: 80.90 -> 73.67 | Hurt |
| Add video-guided VEGAS visual processing | VEGAS · Social-IQ-2.0 supervised MCQ QAV visual contribution | Accuracy change: -0.43% to 9.28% | Helped |
| Add subtitles to video input | VEGAS-generalist · IEMOCAP | Accuracy: 25.5% -> 35.8% | Helped |
| Add audio to video and subtitle input | VEGAS-generalist · IEMOCAP | Accuracy: 35.8% -> 37.4% | Helped |
| Add audio to video and subtitle input | VEGAS-generalist · EmVidCap | GPT-3.5-turbo score: 2.3 -> 1.7 | Hurt |
Surprising findings
- A T5-small language model reached 100% accuracy on Social-IQ-1.0 under BlindQA, indicating strong language bias.
- Full-parameter fine-tuning of the baseline performed worse, while the VEGAS variant reduced language shortcuts further.
Limitations
- Open-ended evaluation relies on GPT-3.5-turbo judging, which may be imprecise.
- Zero-shot open-ended model still struggles with in-depth analysis according to the authors' examples.
Experimental setup
Datasets: Social-IQ-2.0 TVQA Next-QA Video-ChatGPT dataset RAVDESS AudioCaps CMU-MOSEI ExpW ActivityNet-200
Benchmarks: Social-IQ-2.0 IEMOCAP EmVidCap
Metrics: accuracy score (1-5) modality ablation accuracy
Baselines: Video-LLaMA Video-ChatGPT PG-Video-LLaVA Video-LLaVA DeSIQ MMTC-ESC Just-Ask-Plus IVA R-VLM
Builds on: Video-LLaVA LanguageBind Video-ChatGPT DeSIQ Vicuna CLIP
Closest papers by meaning
- Exploiting the Social-Like Prior in Transformer for Visual Reasoning (AAAI 2024 · similarity 0.87)
- Modeling Multimodal Social Interactions: New Challenges and Baselines with Densely Aligned Representations (CVPR 2024 · similarity 0.86)
- Visual Graph Arena: Evaluating Visual Conceptualization of Vision and Multimodal Large Language Models (ICML 2025 · similarity 0.86)
- MimeQA: Towards Socially-Intelligent Nonverbal Foundation Models (NeurIPS 2025 · similarity 0.86)
- HumanSense: From Multimodal Perception to Empathetic Context-Aware Responses Through Reasoning MLLMs (AAAI 2026 · similarity 0.86)
- Video-Only ToM: Enhancing Theory of Mind in Multimodal Large Language Models (CVPR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.