Research map / Video / Video-language understanding
Bringing RNNs Back to Efficient Open-Ended Video Understanding
Weili Xu, Enxin Song, Wenhao Chai, Xuexiang Wen, Tian Ye, Gaoang Wang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| sort merged visual tokens in ascending size order instead of random order (ablation) | AuroraLong · ActivityNet-QA | accuracy: 53.1 -> 56.3 | Improved |
| sort merged visual tokens in ascending order instead of descending (ablation) | AuroraLong · ActivityNet-QA | accuracy: 55.0 -> 56.3 | Improved |
| sort merged visual tokens ascending instead of random (ablation) | AuroraLong · VATEX | BLEU@1: 67.6 -> 68.5 | Improved |
| sort merged visual tokens ascending instead of descending (ablation) | AuroraLong · VATEX | BLEU@1: 67.0 -> 68.5 | Improved |
| sort merged visual tokens ascending instead of random (ablation) | AuroraLong · MovieChat-1K | accuracy: 76.5 -> 78.5 | Improved |
| sort merged visual tokens ascending instead of descending (ablation) | AuroraLong · MovieChat-1K | accuracy: 76.0 -> 78.5 | Improved |
| sort merged visual tokens ascending instead of random (ablation) | AuroraLong · VDC (average) | accuracy: 40.9 -> 41.3 | Improved |
| sort merged visual tokens ascending instead of descending (ablation) | AuroraLong · VDC (average) | accuracy: 41.1 -> 41.3 | Improved |
| lower visual token retention ratio (more merging) for captioning tasks (ablation) | AuroraLong · VATEX and VDC | captioning score | Improved |
| keep token retention at 0.1 instead of higher retention (ablation) | AuroraLong · most video tasks | accuracy | No clear effect |
| Setting B (always merge tokens at ratio 0.1, including single images) instead of Setting A (merge only video samples) (ablation) | AuroraLong · ANet, VATEX, MovieChat, VDC | performance: extra 50% training time for Setting A | Improved |
| replace Transformer LLM with linear RNN (RWKV) backbone and token merge | AuroraLong vs InternVL-1.5 2B · 1024 frames | GPU memory: 34X less memory | Improved |
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 2 of 2 on most shared results.
- AuroraCap (7B) — AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmark (ICLR 2025) · better on 4 of 5 shared results
- Video-XL (7B) — Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understanding (CVPR 2025) · better on 1 of 1 shared result
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- One Token per Highly Selective Frame: Towards Extreme Compression for Long Video Understanding (NeurIPS 2025 · similarity 0.89)
- LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding (ICML 2025 · similarity 0.88)
- Leveraging Textual Memory and Key Frame Reasoning for Full Video Understanding Using Off-the-Shelf LLMs and VLMs (Student Abstract) (AAAI 2025 · similarity 0.87)
- Streaming Long Video Understanding with Large Language Models (NeurIPS 2024 · similarity 0.87)
- Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understanding (CVPR 2025 · similarity 0.87)
- ReWind: Understanding Long Videos with Instructed Learnable Memory (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
