Research map / Vision-language and multimodal / Multimodal LLMs

LatentOmni: Rethinking Omni-Modal Understanding via Unified Audio-Visual Latent Reasoning

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes LatentOmni, which interleaves text reasoning with audio-visual latent states in a unified latent space, trained via SFT on a new 35K interleaved AV reasoning dataset, and reports gains over Qwen2.5-Omni-7B on four omnimodal benchmarks.

Problem

Text-based chain-of-thought in omni-modal MLLMs compresses continuous audio-visual signals into discrete tokens, losing temporal detail and shifting reasoning toward language priors, which hurts joint audio-visual reasoning.

Approach

Interleave discrete text reasoning with continuous audio-visual latent states (last-layer hidden states fed back as inputs), supervised by feature-level latent alignment to pooled audio/visual anchors, a symmetric InfoNCE audio-visual sync loss, and Omni-Sync Position Embedding that shares timestamps across latent audio and visual states.

Key results

• LatentOmni reaches 67.4 on Daily-Omni, 48.9 on WorldSense, 35.4 on OmniVideoBench and 35.1 on LVOmniBench, versus 62.9, 45.4, 29.3 and 32.0 for Qwen2.5-Omni-7B.
• LatentOmni outperforms the Explicit Text CoT baseline by 1.8 (Daily-Omni), 2.3 (WorldSense), 2.2 (OmniVideoBench) and 3.0 (LVOmniBench) points.
• On VideoMME under a vision-only protocol, LatentOmni scores 60.8 overall versus 51.6 for Monet and 36.7 for LVR.

Surprising findings

• Explicit Text CoT on Qwen2.5-Omni-7B improves over the base model on several benchmarks, and Vanilla SFT lowers Daily-Omni accuracy below the base model (62.0 vs 62.9).

Limitations

• Modality coverage limited to audio, visual and text; no 3D, tactile or action signals.
• Gains over open-source baselines are modest on some benchmarks (e.g., +1.8 over Explicit Text CoT on Daily-Omni), and no variance or significance reported.
• Training uses a batch size of 1 with gradient accumulation and only 750 steps, with compute constraints noted by the authors.

Experimental setup

Datasets: ASID-1M AVoCaDO LatentOmni-Instruct-35K

Benchmarks: Daily-Omni WorldSense OmniVideoBench LVOmniBench VideoMME

Metrics: accuracy

Baselines: VideoLLaMA2-7B MiniCPM-o-7B VITA-1.5-7B HumanOmniV2-7B Baichuan-Omni-1.5 OmniVinci Explicit Text CoT Vanilla SFT Monet LVR

Builds on: Coconut Qwen2.5-Omni InfoNCE

Closest papers by meaning

• A Benchmark for Omni-Modal Reasoning in Long Videos (NeurIPS 2026 · similarity 0.91)
• OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMs (ICLR 2026 · similarity 0.91)
• JointAVBench: A Benchmark for Joint Audio-Visual Reasoning Evaluation (ICLR 2026 · similarity 0.91)
• Fill the GAP: A Granular Alignment Paradigm for Visual Reasoning in Multimodal Large Language Models (NeurIPS 2026 · similarity 0.90)
• XModBench: Benchmarking Cross-Modal Capabilities and Consistency in Omni-Language Models (ICLR 2026 · similarity 0.89)
• Visual Enhanced Depth Scaling for Multimodal Latent Reasoning (NeurIPS 2026 · similarity 0.89)