Research map / Video / Video-language understanding
Video-Panda: Parameter-efficient Alignment for Encoder-free Video-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Video-Panda, an encoder-free video-language model using a Spatio-Temporal Alignment Block (STAB) with 45M visual parameters, trained with distillation from LanguageBind on Valley/WebVid and Video-ChatGPT instruction data.
Problem
Video-language models rely on heavyweight image or video encoders (300M-1.4B parameters), creating large computational overhead when processing multi-frame videos.
Approach
A Spatio-Temporal Alignment Block that directly processes video patches with 3D-conv local spatio-temporal encoding, learned-query attention downsampling, and frame-wise plus global video-level aggregators, distilled from LanguageBind features into a Vicuna-7B LLM without any pretrained encoder.
Key results
- Video-Panda reaches 64.7% accuracy on MSVD-QA and 54.8% on MSRVTT-QA with only 45M visual parameters, versus 307M for Video-ChatGPT and 425M for Video-LLaVA.
- On fine-grained video QA, Video-Panda averages 2.57 versus 2.38 for Video-ChatGPT and 2.40 for Video-LLaVA* under same-data training.
- Video-Panda processes videos in 41 ms versus 171 ms for Video-ChatGPT and 125 ms for Video-LLaVA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add <row> tokens to preserve spatial structure (ablation) | Video-Panda · MSVD-QA | accuracy: 63.2 -> 64.7 | Helped |
| add FSRA (ablation) | Video-Panda · MSVD-QA | accuracy: 63.4 -> 64.7 | Helped |
| use learned LSD instead of average pooling (ablation) | Video-Panda · MSVD-QA | accuracy: 58.0 -> 64.7 | Helped |
| add LSTE (ablation) | Video-Panda · MSVD-QA | accuracy: 63.6 -> 64.7 | Helped |
| add GSTRA (ablation) | Video-Panda · MSVD-QA | accuracy: 63.0 -> 64.7 | Helped |
| add LSTE and GSTRA (ablation) | Video-Panda · MSVD-QA | accuracy: 62.2 -> 64.7 | Helped |
| use negative cosine similarity rather than MSE distillation (ablation) | Video-Panda · MSVD-QA | accuracy: 63.5 -> 64.7 | Helped |
| use negative cosine similarity rather than no distillation (ablation) | Video-Panda · MSVD-QA | accuracy: 63.1 -> 64.7 | Helped |
| use negative cosine similarity rather than MSE distillation (ablation) | Video-Panda · ActivityNet-QA | accuracy: 38.2 -> 40.0 | Helped |
| use negative cosine similarity rather than no distillation (ablation) | Video-Panda · ActivityNet-QA | accuracy: 39.8 -> 40.0 | No clear effect |
| use Video-Panda instead of Video-ChatGPT | Vicuna-7B video QA · MSVD-QA | accuracy: 64.9 -> 64.7 | Hurt |
| use Video-Panda instead of Video-ChatGPT | Vicuna-7B video QA · MSVD-QA | score: 3.3 -> 3.8 | Helped |
Surprising findings
- Removing distillation entirely (63.1% on MSVD-QA) still performs close to the full model, suggesting architecture drives much of the performance.
- Video-Panda uses only 8 frames versus 100 for Video-ChatGPT yet outperforms it on several fine-grained metrics.
Limitations
- Fine-grained detail and consistency scores are slightly lower than Video-ChatGPT.
- Evaluated only on 8 frames per video and short-to-medium benchmarks; longer video handling is left for future work.
- Encoder-free model trails stronger encoder-based models trained on much larger data (e.g., ST-LLM, VideoChat2).
Experimental setup
Datasets: WebVid Valley Video-ChatGPT instruction dataset MSRVTT-QA MSVD-QA TGIF-QA ActivityNet-QA ActivityNet-200
Benchmarks: MSVD-QA MSRVTT-QA TGIF-QA ActivityNet-QA Video-ChatGPT fine-grained video QA benchmark
Metrics: accuracy score (1-5) FLOPs latency parameter count
Baselines: Video-ChatGPT Video-LLaVA VideoChat Video-LLaMA ChatUniVi LLaMA-VID VideoChat2 ST-LLM EVE LLaMA Adapter
Builds on: EVE Video-ChatGPT LanguageBind Vicuna Valley
Closest papers by meaning
- Self-alignment of Large Video Language Models with Refined Regularized Preference Optimization (NeurIPS 2025 · similarity 0.90)
- Streaming Long Video Understanding with Large Language Models (NeurIPS 2024 · similarity 0.89)
- Language-aware Visual Semantic Distillation for Video Question Answering (CVPR 2024 · similarity 0.89)
- LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding (NeurIPS 2024 · similarity 0.89)
- Distilling Vision-Language Models on Millions of Videos (CVPR 2024 · similarity 0.88)
- VideoPrism: A Foundational Visual Encoder for Video Understanding (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.