Research map / Video / Video-language understanding
Breaking the Encoder Barrier for Seamless Video-Language Understanding
Handong Li, Yiyuan Zhang, Longteng Guo, Xiangyu Yue, Jing Liu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add cross-attention video patch embedding layer (ablation) | ELVA 7B · VideoMME | accuracy: 39.6 -> 42.2 | Improved |
| add cross-attention video patch embedding layer (ablation) | ELVA 7B · MSVD-QA | accuracy: 48.7 -> 49.9 | Improved |
| use video-pretrained SigLIP as guidance target (ablation) | ELVA 7B · VideoMME | accuracy: 41.0 -> 42.2 | Improved |
| use video-pretrained EVACLIP-g as guidance target (ablation) | ELVA 7B · VideoMME | accuracy: 36.8 -> 38.4 | Improved |
| replace original captions with recaptioned image+video data (ablation) | ELVA 7B · VideoMME | accuracy: 34.2 -> 38.5 | Improved |
| recaption data (ablation) | ELVA 7B · GQA | accuracy: 42.1 -> 46.1 | Improved |
| scale pretraining data from 1M to 7M (ablation) | ELVA 7B · MSVD-QA | accuracy: 47.1 -> 49.4 | Improved |
| scale pretraining data from 1M to 7M (ablation) | ELVA 7B · VideoMME | accuracy: 38.1 -> 38.5 | Mixed |
| add MSE tube-wise alignment loss to generative loss (ablation) | ELVA 7B · GQA | accuracy: 42.2 -> 43.6 | Improved |
| add contrastive frame-wise loss to generative loss (ablation) | ELVA 7B · VideoMME | accuracy: 37.9 -> 38.5 | Improved |
| combine MSE and contrastive losses with generative loss (ablation) | ELVA 7B · SEED-I | accuracy: 40.0 -> 44.8 | Improved |
| add 16 low-res frames to 16 high-res frames at inference (ablation) | ELVA 1.5B · VideoMME | accuracy: 37.1 -> 43.0 | Improved |
Closest papers by meaning
- Unifying Specialized Visual Encoders for Video Language Models (ICML 2025 · similarity 0.88)
- LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding (ICML 2025 · similarity 0.88)
- Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenization (ICML 2024 · similarity 0.88)
- Video-Panda: Parameter-efficient Alignment for Encoder-free Video-Language Models (CVPR 2025 · similarity 0.88)
- Unleashing Hour-Scale Video Training for Long Video-Language Understanding (NeurIPS 2025 · similarity 0.88)
- Enhancing Temporal Understanding in Video-LLMs through Stacked Temporal Attention in Vision Encoders (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
