Research map / Vision-language and multimodal / Multimodal LLMs

OpenOmni: Advancing Open-Source Omnimodal Large Language Models with Progressive Multimodal Alignment and Real-time Emotional Speech Synthesis

NeurIPS 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Unified any-modality language models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes OpenOmni, a fully open-source omnimodal LLM that aligns vision to speech via text-pivoted progressive training and adds a lightweight streaming speech decoder (AR/NAR) tuned with CTC-DPO for emotional speech.

Problem

Open-source omnimodal LLMs lack high-quality tri-modal (image-speech-text) data, rely on slow autoregressive or external TTS speech generation, and cannot produce emotionally expressive speech.

Approach

Use text as a pivot: align speech-text and image-text separately on a pretrained speech LLM so that vision transfers to speech zero-shot, then train a lightweight MoE+transformer speech decoder with CTC for real-time non-autoregressive speech and apply DPO on emotion preference pairs (CTC-DPO).

Key results

• OpenOmni (7B) reaches 37.40 overall on OmniBench vs 33.45 for VITA (7x8B), with about 1.6M vs 5M training samples.
• Non-autoregressive speech generation yields under 1 second latency for up to 30 seconds of speech, over 5x faster than autoregressive methods.
• CTC-DPO raises overall emotion classification accuracy on the Chinese EO2S-9K test set from 57.9 to 70.4 and on English from 62.6 to 65.4.

Measured effects reported in the paper

Surprising findings

• Image-text instruction tuning alone lets the model respond to spoken instructions with near zero-shot transfer, though responses are text-only.
• Freezing the LLM during speech-text alignment performs nearly as well as unfreezing, while image-text alignment needs the LLM unfrozen.

Limitations

• Focused only on Chinese and English; not validated on multilingual data.
• NAR CTC-based training is unstable and quality depends on speech vocabulary size.
• Speech output quality depends on the correctness of LLM text outputs.

Experimental setup

Datasets: WeNetSpeech LibriSpeech AIShell-4 LLaVA-Pretrain-595K MMEvol UltraChat O2S-300K EO2S-9K

Benchmarks: OmniBench AV-Odyssey Bench MMBench-EN MMBench-CN MMStar RealWorldQA MMMU MathVista AI2D HallusionBench AISHELL-2 LibriSpeech

Metrics: accuracy word error rate emotion classification accuracy latency

Baselines: VITA VITA-1.5 Baichuan-Omni EMOVA Freeze-Omni Mini-Omni Qwen2-Audio UnifiedIO2

Builds on: LLaVA Direct Preference Optimization Whisper CosyVoice

Closest papers by meaning

• OmniBench: Towards The Future of Universal Omni-Language Models (NeurIPS 2025 · similarity 0.90)
• EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotions (CVPR 2025 · similarity 0.89)
• Omni-Captioner: Data Pipeline, Models, and Benchmark for Omni Detailed Perception (ICLR 2026 · similarity 0.88)
• VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interaction (NeurIPS 2025 · similarity 0.88)
• Omni-Interactive Universal Embedder (NeurIPS 2026 · similarity 0.87)
• Dynamic-SUPERB Phase-2: A Collaboratively Expanding Benchmark for Measuring the Capabilities of Spoken Language Models with 180 Tasks (ICLR 2025 · similarity 0.87)