Research map / Vision-language and multimodal / Multimodal LLMs
VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interaction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Presents VITA-1.5, an open multimodal LLM that adds speech input and end-to-end speech output to vision-language capabilities via a three-stage progressive training pipeline, avoiding separate ASR/TTS modules.
Problem
Integrating vision and speech into one MLLM is hard because the modalities conflict during training (speech data can degrade vision and vice versa), and cascaded ASR-LLM-TTS speech systems add latency and lose paralinguistic cues.
Approach
A three-stage progressive training recipe (vision-language, then audio input, then audio output with a frozen LLM) that introduces modalities incrementally to limit cross-modal interference, plus an in-model NAR+AR speech decoder that generates speech tokens from the LLM's text embeddings for end-to-end speech output.
Key results
- On image benchmarks VITA-1.5 (Stage 1) averages 67.1 across nine benchmarks, close to VITA-1.5-Audio (66.8) and above GPT-4V (58.5) and GPT-4o mini (66.3), but below GPT-4o (69.3).
- On ASR, VITA-1.5 reaches 2.2 CER on aishell-1 and 3.4 WER on LibriSpeech test-clean, versus 4.1 for Freeze-Omni and 4.7 for Mini-Omni2.
- On video, VITA-1.5 (Stage 3) scores 56.1 on Video-MME without subtitles and 66.7 on TempCompass, trailing GPT-4o (71.9 and 73.8).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add audio input and output tuning after Stage 1 vision-language training (ablation) | VITA-1.5 · Image benchmarks | Avg: 67.1 -> 66.8 | Hurt |
| Add audio input and output tuning after Stage 1 vision-language training (ablation) | VITA-1.5 · MMBench | score: 77.1 -> 76.7 | Hurt |
| Add audio input and output tuning after Stage 1 vision-language training (ablation) | VITA-1.5 · MMStar | score: 59.1 -> 59.9 | Helped |
| Add audio input and output tuning after Stage 1 vision-language training (ablation) | VITA-1.5 · MMMU | score: 53.1 -> 52.1 | Hurt |
| Add audio input and output tuning after Stage 1 vision-language training (ablation) | VITA-1.5 · MathVista | score: 66.2 -> 66.2 | No clear effect |
| Add audio input and output tuning after Stage 1 vision-language training (ablation) | VITA-1.5 · HallusionBench | score: 44.1 -> 44.9 | Helped |
| Add audio input and output tuning after Stage 1 vision-language training (ablation) | VITA-1.5 · AI2D | score: 80.3 -> 79.3 | Hurt |
| Add audio input and output tuning after Stage 1 vision-language training (ablation) | VITA-1.5 · OCRBench | score: 752.0 -> 732.0 | Hurt |
| Add audio input and output tuning after Stage 1 vision-language training (ablation) | VITA-1.5 · MMVet | score: 51.1 -> 49.6 | Hurt |
| Add audio input and output tuning after Stage 1 vision-language training (ablation) | VITA-1.5 · MME | score: 2311.0 -> 2352.0 | Helped |
| Add audio input and output tuning after Stage 1 vision-language training (ablation) | VITA-1.5 · Video-MME w/o sub | score: 56.8 -> 56.1 | Hurt |
| Add audio input and output tuning after Stage 1 vision-language training (ablation) | VITA-1.5 · Video-MME w/ sub | score: 59.5 -> 58.7 | Hurt |
Surprising findings
- Adding speech output in Stage 3 barely changes vision scores, and the frozen LLM keeps multimodal performance intact.
- The authors note a notable gap to proprietary models on video understanding, indicating room for improvement.
Limitations
- Video understanding lags well behind GPT-4o and Gemini 1.5 Pro.
- Model is generic and does not personalize to user preferences.
- No long-term memory for extended multi-hour interaction.
Experimental setup
Datasets: ShareGPT4V ALLaVA-Caption ShareGPT4o-Image LLaVA-150K LLaVA-Mixture-sample LVIS-Instruct ScienceQA ChatQA LLaVA-OV Anyword-3M ICDAR2019-LSVT UReader
Benchmarks: MME MMBench MMStar MMMU MathVista HallusionBench AI2D OCRBench MMVet Video-MME MVBench TempCompass
Metrics: accuracy average score CER WER
Baselines: VILA-1.5 LLaVA-Next CogVLM2 InternVL-2 MiniCPM-V-2.6 GPT-4o Gemini 1.5 Pro Mini-Omni2 Freeze-Omni Wav2vec2-base
Builds on: VITA-1.0 LLaVA-OneVision InternViT CTC
Closest papers by meaning
- Vision-Speech Models: Teaching Speech Models to Converse about Images (CVPR 2026 · similarity 0.90)
- HAVE-Bench: Hierarchical Audio-Visual Evaluation from Perception to Interaction (CVPR 2026 · similarity 0.88)
- VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Tasks (NeurIPS 2024 · similarity 0.88)
- Multi-modal Instruction Tuned LLMs with Fine-grained Visual Perception (CVPR 2024 · similarity 0.88)
- CoVFT: Context-aware Visual Fine-tuning for Multimodal Large Language Models (CVPR 2026 · similarity 0.88)
- MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.