Research map / Video / Video-language understanding

Watch and Listen: Understanding Audio-Visual-Speech Moments with Multimodal LLM

NeurIPS 2025 · Video › Video-language understanding · approach: Video LLMs with fine-grained understanding · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes TriSense, a triple-modality (visual, audio, speech) MLLM with a query-based connector that reweights modalities per query, and TriSense-2M, a 2M-sample omni-modal temporal dataset built with fine-tuned Qwen2.5-72B generator/judger.

Problem

Video temporal understanding (moment retrieval and segment captioning) with audio, speech, and visual modalities, robust to missing or noisy modalities and long videos.

Approach

A Query-Based Connector computes query-conditioned cross-attention features per modality, pools them into global vectors, and applies a softmax-weighted fusion so the LLM emphasizes relevant modalities; combined with modality dropout in training and a time encoder/head with a sync token for timestamps.

Key results

• On TriSense-2M AVS moment retrieval, TriSense reaches R@0.5 1.12 and R@0.7 0.42 versus 0.61 and 0.21 for Qwen2.5-Omni (7B).
• On zero-shot ActivityNet-Captions, TriSense achieves R@0.7 of 27.2 versus 24.7 for TRACE-uni (7B).
• On Video-MME, TriSense scores 48.7 versus 49.6 for TRACE-uni, using about 55% of its general understanding data.

Measured effects reported in the paper

Surprising findings

• LongVALE (7B) scores near zero on TriSense-2M moment retrieval (e.g., 0.08 R@0.5 on AVS-MR) despite being an omni-modal model.
• Simply fixing visual weight to 1 gives slightly better visual-only segment captioning than the learned connector.

Limitations

• Visual-only moment retrieval is slightly weaker than state-of-the-art vision models.
• Using only 64 test frames hurts long-video moment retrieval; accuracy improves with 128 frames.
• Training requires substantial compute (16 A100 GPUs for multi-day stages).

Experimental setup

Datasets: TriSense-2M Clotho LLaVA-LCS558K Valentini-Botinhao Speech Dataset LLaVA-Video-178K InternVid VAST Charades-STA ActivityNet-Captions LongVALE

Benchmarks: TriSense-2M LongVALE Charades-STA ActivityNet-Captions Video-MME

Metrics: Recall@IoU=0.5 Recall@IoU=0.7 mIoU BLEU-4 METEOR ROUGE-L CIDEr

Baselines: VTimeLLM TimeChat VTG-LLM TRACE TRACE-uni LongVALE Qwen2.5-Omni Momentor HawkEye NumPro-FT

Builds on: TRACE Mistral-7B LongVALE Slot-Based Compression

Closest papers by meaning

• Empowering LLMs with Pseudo-Untrimmed Videos for Audio-Visual Temporal Understanding (AAAI 2025 · similarity 0.89)
• JointAVBench: A Benchmark for Joint Audio-Visual Reasoning Evaluation (ICLR 2026 · similarity 0.88)
• Aligned Better, Listen Better for Audio-Visual Large Language Models (ICLR 2025 · similarity 0.88)
• Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis (CVPR 2025 · similarity 0.88)
• FAVE: A Structured Benchmark for Fine-Grained Audio-Visual Temporal Evaluation in Multimodal LLMs (CVPR 2026 · similarity 0.88)
• MMOU: A Massive Multi-Task Omni Understanding and Reasoning Benchmark for Long and Complex Real-World Videos (NeurIPS 2026 · similarity 0.87)