Research map / Large language models / Language, safety and interpretability
EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds EMOVA, an end-to-end omni-modal LLM (Qwen-2.5 based) that sees, hears and speaks with emotional/pitch style control, using text-centric alignment of bi-modal data and a semantic-acoustic disentangled speech tokenizer.
Problem
Open-source LLMs lack end-to-end omni-modal (image, text, speech) capability; existing vision-language models use external speech tools, speech LLMs lack vision or speech style control, and omni-modal data is scarce.
Approach
Use publicly available bi-modal image-text and speech-text data aligned jointly with text as a bridge, plus a semantic-acoustic disentangled discrete speech tokenizer and a lightweight style module in the unit-to-speech decoder that lets the LLM predict emotion and pitch labels alongside speech units.
Key results
- EMOVA-72B reaches 83.2 on InfoVQA and 88.7 on ChartQA; EMOVA outperforms GPT-4o/4V and Gemini Pro 1.5 on 11 of 15 vision-language benchmarks.
- EMOVA-7B achieves 814 on OCRBench vs 678 for VITA and 2317 on MME; LibriSpeech WER 2.9 for 72B vs 3.0 for Whisper-Large.
- Joint omni-modal alignment outperforms VL-only, Speech-only and sequential alignment on both vision-language and ASR benchmarks; style label classification accuracy exceeds 75%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Joint omni-modal alignment (image-text plus speech-unit-text trained together) instead of single bi-modal alignment (ablation) | EMOVA alignment paradigm · vision-language and LibriSpeech ASR | vision-language benchmarks and 1-WER | Helped |
| Joint alignment instead of sequential alignment (VL→Speech or Speech→VL) (ablation) | EMOVA alignment paradigm · vision-language and ASR | vision-language and 1-WER | Helped |
| Semantic-acoustic disentangled speech tokenizer instead of HuBERT-based entangled units (ablation) | EMOVA joint alignment · vision-language and ASR | vision-language benchmarks and 1-WER | Helped |
| Omni-modal joint alignment with text-centric bridging, without omni-modal image-text-speech data | EMOVA-7B vs VITA on OCRBench · OCRBench | score: +136 (814 vs 678) | Helped |
| Explicit style label prediction (emotion and pitch) with a style-conditioned U2S detokenizer | EMOVA-7B spoken dialogue · Emotion style classification | accuracy | Helped |
| Higher ASR quality of speech input (ground-truth transcript vs speech units) on dialogue | EMOVA-7B spoken dialogue on Speech-Text sets · Speech-Text-EN and Speech-Text-ZH | dialogue score | Helped |
Surprising findings
- Adding speech-unit-text data improves vision-language benchmarks compared with image-text-only alignment, contrary to the assumption that modalities compete.
- Sequential alignment (VL then speech or speech then VL) underperforms joint alignment, attributed to catastrophic forgetting.
Limitations
- Emotional dialogue evaluation relies on a self-generated test split from synthetic data and GPT-4o judging rather than human-recorded speech.
- ASR performance on complex speech instructions is weaker (WER 6.9 and 10.7 on some dialogue sets), and Chinese TTS error is high (12%).
- Speech style training data is TTS-synthesized, limiting real-life style diversity.
Experimental setup
Datasets: LCS-558K ShareGPT4V ALLaVA ShareGPT-4o SynthDog MMC-Alignment UReader LibriSpeech AISHELL-2 Magpie Pro LibriTTS AISHELL-1
Benchmarks: MME MMBench SEED-Image MM-Vet RealWorldQA TextVQA ChartQA DocVQA InfoVQA OCRBench AI2D ScienceQA-Img
Metrics: accuracy WER CER dialogue score (0-10) style label accuracy
Baselines: GPT-4V GPT-4o Gemini Pro 1.5 VITA-1.5 VITA-8x7B Baichuan-Omni-7B Mini-Omni2 Whisper-Large AnyGPT
Builds on: LLaVA VITS SPIRAL Qwen-2.5 ShareGPT4V
Closest papers by meaning
- OpenOmni: Advancing Open-Source Omnimodal Large Language Models with Progressive Multimodal Alignment and Real-time Emotional Speech Synthesis (NeurIPS 2025 · similarity 0.89)
- VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interaction (NeurIPS 2025 · similarity 0.86)
- OmniBench: Towards The Future of Universal Omni-Language Models (NeurIPS 2025 · similarity 0.86)
- AVI-Bench: Toward Human-like Audio-Visual Intelligence of Omni-MLLMs (ICML 2026 · similarity 0.85)
- Omni-Captioner: Data Pipeline, Models, and Benchmark for Omni Detailed Perception (ICLR 2026 · similarity 0.85)
- XModBench: Benchmarking Cross-Modal Capabilities and Consistency in Omni-Language Models (ICLR 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.