Research map / Video / Video-language understanding
AVOC: Enhancing Hour-Level Audio-Video Understanding in Omni-Modal LLMs via Retrieval-Inspired Token Compression
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AVOC, a learnable retrieval-inspired token compression module (relevance, importance, temporal-aware MMR diversity) inserted into an omni-modal LLM (MiniCPM-o 4.5) to enable hour-level audio-video understanding under a fixed token budget.
Problem
Long-form (hour-level) audio-video understanding is limited by LLM context windows and severe redundancy in audio-video token streams; existing omni-modal compression methods use rigid asymmetric video-audio guidance.
Approach
Reframes multimodal token compression as top-K retrieval and instantiates three IR criteria: text-guided cross-attention for query relevance, bidirectional video-audio cross-attention for query-agnostic importance, and Temporal-Aware MMR (similarity penalized only within a local temporal window) for diversity, with modality-split budgets and Gumbel-Softmax top-K training.
Key results
- AVOC reaches 42.3% average on OmniVideoBench (+4.9 over second-best) and 40.3% average on LVOmniBench (+5.5 over second-best).
- AVOC achieves 52.6% on WorldSense, +1.7 over video-SALMONN 2+ 7B.
- Prefill latency drops from 4.453 s (MiniCPM-o 4.5) to 0.497 s at token retention 0.1 on a 10-minute video, with AVOC maintaining high AV-NIAH accuracy up to 3600 s.
Surprising findings
- Random token selection at the same budget drops OmniVideoBench average only modestly from 42.3 to 37.2 while the paper argues which tokens are kept matters more than how many.
- MiniCPM-o 4.5 backbone collapses on AV-NIAH beyond about 300 seconds, showing the base model's context limit is the bottleneck.
Limitations
- Offline operation: the full audio-video stream must be available before scoring, so streaming use is not supported.
- Experiments only at a single relatively small parameter scale (7-8B class), so scaling is unverified.
- Modality budget ratio is a fixed hyperparameter rather than content-adaptive.
Experimental setup
Datasets: AVSD How2 FineVideo ChronusAV LongVILA_sft LVOmniBench (haystack source)
Benchmarks: WorldSense OmniVideoBench LVOmniBench AV-NIAH
Metrics: accuracy exact-match accuracy
Baselines: VideoLLaMA2-7B Baichuan-Omni-1.5 HumanOmni-7B Qwen2.5-Omni-7B video-SALMONN 2+ 7B MiniCPM-o 2.6 MiniCPM-o 4.5 OmniZip
Builds on: MiniCPM-o 4.5 Maximal Marginal Relevance Gumbel-Softmax OmniZip
Closest papers by meaning
- OmniSelect: Dynamic Modality-Aware Token Compression for Efficient Omni-modal Large Language Models (NeurIPS 2026 · similarity 0.91)
- One Token per Highly Selective Frame: Towards Extreme Compression for Long Video Understanding (NeurIPS 2025 · similarity 0.88)
- ALMTokenizer: A Low-bitrate and Semantic-rich Audio Codec Tokenizer for Audio Language Modeling (ICML 2025 · similarity 0.88)
- video-SALMONN: Speech-Enhanced Audio-Visual Large Language Models (ICML 2024 · similarity 0.88)
- LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding (ICML 2025 · similarity 0.87)
- JointAVBench: A Benchmark for Joint Audio-Visual Reasoning Evaluation (ICLR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.