Research map / Video / Video-language understanding
PAVE: Patching and Adapting Video Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PAVE, a lightweight "patching" framework that adds small adapter modules (side-channel patches with cross-attention, LoRA) to pre-trained Video-LLMs so they can process audio, 3D camera pose, and high-frame-rate video without modifying base weights.
Problem
Pre-trained video LLMs cannot natively use side-channel signals (audio, camera pose, high frame rate video, multi-view video), and full retraining or architecture changes for each new modality are costly.
Approach
Insert small "patches" (temporally aligned cross-attention adapters with a few parameters and LoRA on the LLM) that fuse side-channel tokens with original video tokens while keeping the base model's architecture and pre-trained weights intact, enabling efficient adaptation across tasks and base models.
Key results
- On the AVSD/Music-AVQA audio-visual QA benchmarks, PAVE improves over the zero-shot base Video-LLM with only a small number of added parameters (about 0.1% of total parameters and FLOPs).
- On 3D QA (ScanQA/SQA3D), PAVE with the LLaVA-OneVision base reaches competitive scores against task-specific SOTA models with 3D point cloud-style inputs.
- Adding high-frame-rate side-channel patches allows PAVE to handle high frame-rate video understanding with reduced cost compared to naively feeding more frames.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add an audio patch | LLaVA-OneVision-7B · AVQA | accuracy: 90.8 -> 93.8 | Helped |
| Add an audio patch | LLaVA-OneVision-7B · AVSD | CIDEr: +44 | Helped |
| Add audio to the video-only model | LLaVA-OneVision-7B · Music-AVQA visual split | accuracy: 75.4 -> 79.7 | Helped |
| Add audio and 3D-information patches | LLaVA-OneVision-7B · Music-AVQA audio and audio-visual splits | accuracy | Mixed |
| Add a 3D-information patch | LLaVA-OneVision-7B · SQA3D | top-1 exact match | Helped |
| Add a high-frame-rate video patch | LLaVA-OneVision-7B · VideoMME short and median splits | accuracy: 1.0-4.4% | Helped |
| Add a high-frame-rate video patch | LLaVA-OneVision-7B · VideoMME long split | accuracy: 0.1-0.3% | Helped |
| Use original visual tokens as the cross-attention query instead of learnable tokens (ablation) | PAVE on ScanQA · ScanQA | top-1 exact match: 22.5 (39.3) -> 23.1 (40.0) | Helped |
| Use temporal-aligned cross-attention and token summation instead of interleaving tokens (ablation) | PAVE on ScanQA · ScanQA | top-1 exact match: 23.0 (39.9) -> 23.1 (40.0) | Helped |
| Add a PAVE patch rather than fine-tuning the base video model alone (ablation) | LLaVA-OneVision-0.5B on ScanQA · ScanQA | top-1 exact match: 20.5 (36.3) -> 23.1 (40.0) | Helped |
| Jointly train audio and high-frame-rate patches (ablation) | PAVE-7B · AVSD test | CIDEr: +7.5 | Helped |
Surprising findings
- PAVE's fine-tuned audio-visual variant reduces hallucination-like errors less than expected on pure audio questions, indicating limited transfer to audio-only tasks.
Limitations
- Audio-only and pure side-channel benefits depend on whether the base video model already attends to the video content; PAVE adds little for audio-only tasks.
- Demonstrated mainly on a limited set of side-channel modalities and tasks.
- Patches are trained per task/modality, so multi-task generalization across patches is not fully established.
Experimental setup
Datasets: AVSD Music-AVQA ScanQA SQA3D Star-like AVQA benchmark VideoChat-type instruction data ActivityNet-QA
Benchmarks: AVSD Music-AVQA ScanQA SQA3D MSVD-QA ActivityNet-QA
Metrics: accuracy CIDEr BLEU-4 METEOR ROUGE-L EM@1
Baselines: LLaVA-OneVision (zero-shot) Video-ChatGPT Video-LLaVA VideoChat2 Qwen2-VL Chat-UniVi Video-LLaMA-2 SALMONN
Builds on: LLaVA-OneVision LoRA Video-LLaMA LLaVA-NeXT-Video
Closest papers by meaning
- Enhancing Temporal Understanding in Video-LLMs through Stacked Temporal Attention in Vision Encoders (NeurIPS 2025 · similarity 0.87)
- MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understanding (CVPR 2024 · similarity 0.87)
- BOLT: Boost Large Vision-Language Model Without Training for Long-form Video Understanding (CVPR 2025 · similarity 0.87)
- A Simple Recipe for Contrastively Pre-training Video-First Encoders Beyond 16 Frames (CVPR 2024 · similarity 0.87)
- Understanding Long Videos with Multimodal Language Models (ICLR 2025 · similarity 0.87)
- MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.