Research map / Video / Video-language understanding
Empowering LLMs with Pseudo-Untrimmed Videos for Audio-Visual Temporal Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds PU-VALOR, a 114k-video pseudo-untrimmed audio-visual dataset synthesized from VALOR-32K via caption clustering, random temporal scaling and permutation, and fine-tunes an audio-visual LLM (AVicuna) with an audio-visual token interleaver for temporal localization and AV QA.
Problem
MLLMs lack temporal grounding for audio-visual events because no untrimmed audio-visual video datasets with precise event timestamps and captions exist.
Approach
Synthesize pseudo-untrimmed videos by clustering semantically similar trimmed VALOR clips, randomly time-scaling and concatenating them, and deriving timestamps automatically; then interleave audio and video tokens with an Audio-Visual Token Interleaver before multi-stage LoRA fine-tuning of Vicuna-7B.
Key results
- AVEDL on UnAV-100: AVicuna average mAP 60.3 vs 51.7 for the best prior non-LLM baseline UniAV-ST, with mAP@0.5 of 60.0.
- Open-ended video QA: AVicuna scores 70.2 on MSVD-QA, 59.7 on MSRVTT-QA and 53.0 on ActivityNet-QA, above all compared LLM-based models.
- Removing PU-VALOR drops AVEDL average mAP from 60.3 to 27.9.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add PU-VALOR for Time-Event Alignment (ablation) | AVicuna · UnAV-100 | average mAP: 27.9 -> 60.3 | Helped |
| add the Audio-Visual Token Interleaver (AVTI) (ablation) | AVicuna · UnAV-100 | average mAP: 51.1 -> 60.3 | Helped |
| add A5-222K Audio-Text Alignment data (ablation) | AVicuna · UnAV-100 | average mAP: 30.1 -> 60.3 | Helped |
| include audio input (ablation) | AVicuna · UnAV-100 | average mAP: 35.8 -> 60.3 | Helped |
| set AIR to 25% rather than 0% (ablation) | AVicuna · UnAV-100 | average mAP: 35.8 -> 60.3 | Helped |
| increase AIR from 25% to 40% (ablation) | AVicuna · UnAV-100 | average mAP: 60.3 -> 58.3 | Hurt |
| use AVicuna instead of the strongest reported baseline | video QA and AVQA models · AVSD, MUSIC-AVQA, MSVD-QA, MSRVTT-QA, ActivityNet-QA | score | Helped |
| use AVicuna instead of UniAV-ST | AVEDL models · UnAV-100 | average mAP: 51.7 -> 60.3 | Helped |
| use AVicuna instead of VTimeLLM | VTG models · ActivityNet Captions | R1@0.5: 23.9 -> 25.0 | Helped |
Surprising findings
- Without the PU-VALOR time-event alignment data, AVicuna's AVEDL average mAP collapses to 27.9, far below the non-LLM baselines.
- Audio-interleaving rate has a non-monotonic effect: performance peaks at 25% and drops sharply at 50%, with excessive audio described as detrimental.
Limitations
- Hallucination of plausible but incorrect details.
- Spatial-temporal grounding in long or untrimmed videos is weak.
- Precision is limited for ultra-long videos because time is represented by 100 discrete percentage tokens.
Experimental setup
Datasets: PU-VALOR VALOR-32K LCS-558K A5-222K AudioSet AudioCap Auto-ACD InternVid UnAV-100 VideoInstruct100K ActivityNet Captions DiDeMo
Benchmarks: MSVD-QA MSRVTT-QA ActivityNet-QA AVSD MUSIC-AVQA UnAV-100 ActivityNet Captions
Metrics: GPT score accuracy mAP at tIoU 0.5-0.9 average mAP R1@0.5 R1@0.7 mIoU GPT-based generative scores
Baselines: Valley VideoChat Video-ChatGPT VTimeLLM PandaGPT Macaw-LLM AV-LLM Video-LLaMA VSGN TadTR ActionFormer UnAV
Builds on: Vicuna VTimeLLM LLaVA VALOR CLIP CLAP
Closest papers by meaning
- Aligned Better, Listen Better for Audio-Visual Large Language Models (ICLR 2025 · similarity 0.91)
- Dense Audio-Visual Event Localization Under Cross-Modal Consistency and Multi-Temporal Granularity Collaboration (AAAI 2025 · similarity 0.91)
- FAVE: A Structured Benchmark for Fine-Grained Audio-Visual Temporal Evaluation in Multimodal LLMs (CVPR 2026 · similarity 0.90)
- VTimeLLM: Empower LLM to Grasp Video Moments (CVPR 2024 · similarity 0.90)
- Enhancing Temporal Understanding in Video-LLMs through Stacked Temporal Attention in Vision Encoders (NeurIPS 2025 · similarity 0.90)
- TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.