Research map / Video / Video-language understanding

LongVALE: Vision-Audio-Language-Event Benchmark Towards Time-Aware Omni-Modal Perception of Long Videos

CVPR 2025 · Video › Video-language understanding · approach: Video LLMs with fine-grained understanding · 6 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Builds LongVALE, an automatically annotated omni-modal (vision, audio, speech) long-video benchmark with event boundaries and correlation-aware captions, and a Video-LLM baseline (LongVALE-LLM) trained on it.

Problem

Existing video benchmarks lack fine-grained, time-bounded event annotations that combine visual, audio and speech information on long videos, and manual annotation of such data is costly.

Approach

An automatic pipeline that filters long videos for audio-visual correspondence, detects omni-modal event boundaries using both visual scene cuts and audio (MFCC-delta plus CLAP merging), and generates correlation-aware omni-modal captions with Gemini-1.5-Pro, followed by manual refinement of the test set; a video LLM with audio and speech adapters is trained on the result.

Key results

• LongVALE-LLM (7B) reaches 15.7 R@0.3 and 22.4 mIoU... with Omni-TVG mIoU 5.6 and Omni-DVC CIDEr 20.3 (easy/hard split), clearly above VTimeLLM and TimeChat baselines.
• Zero-shot, LongVALE-LLM achieves 54.8 on AVSD, state of the art among compared methods, using 0.7M pairs versus up to 1007M for OneLLM.
• Omitting audio-visual correlation reasoning in captions lowers Omni-SC CIDEr from 21.1 to 10.4 on the easy subset.

Measured effects reported in the paper

Surprising findings

• Zero-shot AVQA performance on AVSD exceeds models trained on far more audio-visual data, using only about 32.7K samples from LongVALE.
• Instruction tuning on high-quality dialogues improves segment captioning consistently, despite boundary-perception tuning already yielding strong results.

Limitations

• Captions depend on closed API models (Gemini-1.5-Pro, GPT-4o), with possible hallucinations despite refinement.
• Test set is manually checked but training set is fully automatic.
• Video LLM uses only 100 sampled frames and per-frame CLS features, limiting temporal and visual detail.

Experimental setup

Datasets: ACAV-100M LongVALE LCS-558k AVSD Music-AVQA VTimeLLM visual-only data

Benchmarks: LongVALE test set AVSD Music-AVQA

Metrics: Recall@1 at IoU 0.3/0.5/0.7 mIoU SODA_c CIDEr METEOR BLEU-4 ROUGE-L accuracy

Baselines: VideoChat VideoChatGPT VideoLLaMA PandaGPT NExT-GPT TimeChat VTimeLLM Macaw-LLM X-InstructBLIP AV-LLM OneLLM AVicuna

Builds on: VTimeLLM ACAV-100M C-MCR CLAP Vicuna

Closest papers by meaning

• A Benchmark for Omni-Modal Reasoning in Long Videos (NeurIPS 2026 · similarity 0.91)
• MMOU: A Massive Multi-Task Omni Understanding and Reasoning Benchmark for Long and Complex Real-World Videos (NeurIPS 2026 · similarity 0.90)
• JointAVBench: A Benchmark for Joint Audio-Visual Reasoning Evaluation (ICLR 2026 · similarity 0.90)
• OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMs (ICLR 2026 · similarity 0.89)
• MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models (CVPR 2025 · similarity 0.89)
• Dense Audio-Visual Event Localization Under Cross-Modal Consistency and Multi-Temporal Granularity Collaboration (AAAI 2025 · similarity 0.89)