Research map / Vision-language and multimodal / Multimodal LLMs
Can Large Language Models Help Multimodal Language Analysis? MMLA: A Comprehensive Benchmark
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MMLA, a 61K-utterance benchmark across six multimodal language analysis dimensions, and evaluates LLMs and MLLMs under zero-shot, SFT, and instruction tuning with LoRA.
Problem
Existing MLLM benchmarks focus on low-level perception and do not assess cognitive-level semantics (intent, emotion, dialogue act, sentiment, speaking style, communication behavior) in conversational utterances.
Approach
A unified benchmark aggregating nine multimodal language datasets into six dimensions, evaluating LLMs and MLLMs across zero-shot inference, per-task supervised fine-tuning with LoRA, and multi-task instruction tuning.
Key results
- Zero-shot GPT-4o is the best MLLM at 52.6% average ACC, while the best fine-tuned model (72B Qwen2-VL, SFT) reaches 69.18% average ACC.
- After SFT, 7B MLLMs reach 67.47–68.30% ACC versus 68.44–69.18% for 72B counterparts, a gap of only 1–2%.
- After instruction tuning, 72B Qwen2-VL reaches 86.3% on MIntRec, exceeding the reported human score of 85.5%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply supervised fine-tuning rather than zero-shot inference | foundation models · Intent | ACC: +20~40% | Helped |
| apply supervised fine-tuning rather than zero-shot inference | foundation models · Dialogue Act | ACC: +10~40% | Helped |
| apply supervised fine-tuning rather than zero-shot inference | foundation models · Speaking Style | ACC: +4~20% | Helped |
| apply supervised fine-tuning rather than zero-shot inference | foundation models · Communication Behavior | ACC: +5~50% | Helped |
| use 8B MiniCPM-V-2.6 instead of larger MLLMs after SFT | MMLA combined test sets · MMLA combined test sets | average ACC: 68.88%, only 0.3% lower in ACC than the top model | Helped |
| use instruction tuning instead of SFT | LLaVA-Video-72B · MMLA combined test sets | average ACC: +0.43% | Helped |
| use instruction tuning instead of SFT | Qwen2-VL · MMLA combined test sets | average ACC: 0.54% for 72B and 0.26% for 7B | Hurt |
| use instruction tuning instead of SFT | MiniCPM-V-2.6 · MMLA combined test sets | average ACC: 1.63% | Hurt |
| use 72B Qwen2-VL with instruction tuning | MMLA · MIntRec | ACC: 86.3% vs. 85.5% | Helped |
| scale Qwen2 from 0.5B to 1.5B parameters | Qwen2 · MMLA dimensions | ACC | Mixed |
| scale Qwen2 from 1.5B to 7B parameters | Qwen2 · MMLA dimensions | ACC | Mixed |
| scale Qwen2-VL from 7B to 72B parameters after SFT | Qwen2-VL · Speaking Style and Intent | ACC: over 5% | Helped |
Surprising findings
- Small 7–8B MLLMs after SFT or IT rival 72B models, with parameter scale mattering little once fine-tuned.
- Some MLLMs (LLaVA-Video, LLaVA-OV) produce irrelevant outputs with severe hallucinations after instruction tuning and are omitted.
Limitations
- Fine-tuned models still reach only about 60–70% accuracy, leaving substantial headroom.
- GPT-4o required iterative prompt modification to force valid label outputs, so zero-shot results depend on prompt engineering.
- Some datasets (e.g., MOSI, IEMOCAP, Anno-MI) restrict video redistribution, so users must obtain videos from official sources.
Experimental setup
Datasets: MIntRec MIntRec2.0 MELD IEMOCAP MELD-DA IEMOCAP-DA MOSI CH-SIMS v2.0 UR-FUNNY-v2 MUStARD Anno-MI
Benchmarks: MMLA MIntRec MIntRec2.0 MELD IEMOCAP MOSI CH-SIMS v2.0 UR-FUNNY-v2 MUStARD Anno-MI
Metrics: ACC WF1 WP F1 recall precision
Baselines: GPT-4o MIntOOD MulT UniMSE TCL-MAP MMML ALMT SimMMDG
Builds on: LoRA LLaMAFactory FlashAttention-2 LLaVA-OneVision Qwen2-VL
Closest papers by meaning
- MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models (NeurIPS 2025 · similarity 0.92)
- SEED-Bench: Benchmarking Multimodal Large Language Models (CVPR 2024 · similarity 0.91)
- Debiased Multimodal Understanding for Human Language Sequences (AAAI 2025 · similarity 0.91)
- On Path to Multimodal Generalist: General-Level and General-Bench (ICML 2025 · similarity 0.90)
- Beyond task performance: evaluating and reducing the flaws of large multimodal models with in-context-learning (ICLR 2024 · similarity 0.90)
- LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models (ICLR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.