AI Paper Atlas

Research map / Video / Video-language understanding

InternVideo2: Scaling Foundation Models for Multimodal Video Understanding

Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Jilan Xu, Zun Wang, Yansong Shi, Tianxiang Jiang and 6 more

ECCV 2024 · Video › Video-language understanding · approach: Video LLMs with fine-grained understanding · 438 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
scale video encoder from 1B to 6B (zero-shot action recognition average) (ablation)InternVideo2-s2 · zero-shot action recognition averageaccuracy (%): 55.5 -> 56.9Improved
scale video encoder from 1B to 6B (finetuned action recognition) (ablation)InternVideo2-s1 · finetuned action recognition averageaccuracy (%): 73.2 -> 73.6No clear effect
add multimodal stage 2 pretraining before linear probing (ablation)InternVideo2-6B · Something-Something V2 linear probingtop-1 accuracy (%): 47.8 -> 56.7Improved
use AutoShot semantic segmentation instead of SceneDet filter (ablation)InternVideo2 stage 2 data · MSR-VTT zero-shot T2VR@1: 27.1 -> 34.8Improved
add audio captions to video captions (ablation)InternVideo2 stage 2 data · MSR-VTT zero-shot T2VR@1: 24.7 -> 26.6Improved
add speech captions to video and audio captions (ablation)InternVideo2 stage 2 data · MSR-VTT zero-shot T2VR@1: 26.6 -> 27.1Improved
add audio encoder trained with video-audio-text learning (ablation)stage 2 baseline · MSR-VTT zero-shot T2V (ViT-B)R@1: 24.7 -> 27.8Improved
add both audio and speech encoders (ablation)stage 2 baseline · MSR-VTT zero-shot T2V (ViT-B)R@1: 27.8 -> 25.7Mixed
distill from both multimodal and motion-aware teachers instead of multimodal teacher only (ablation)stage 1 ViT-1B · SSv2 finetunedtop-1 accuracy (%): 76.5 -> 77.1Improved
enlarge pretraining data from K710 to K-Mash 1.1M (ablation)stage 1 ViT-1B · finetuned action recognition averageaccuracy (%): 78.5 -> 79.0Improved
inject questions into the QFormer during stage 3 instruction tuning (ablation)VideoChat2 with InternVideo2 · Egoschema-fullaccuracy (%): 55.8 -> 52.9Hurt
scale ViT from 1B to 6B for fine-grained localization (ablation)InternVideo2-s1 · FineAction TALavg mAP: 27.2 -> 27.7Mixed

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy