AI Paper Atlas

Research map / Video / Video-language understanding

Scaling Multimodal Pre-Training via Cross-Modality Gradient Harmonization

Junru Wu, Yi Liang, feng han, Hassan Akbari, Zhangyang Wang, Cong Yu

NeurIPS 2022 · Video › Video-language understanding · approach: Video-language pretraining · 6 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add gradient realignment and gradient-based curriculum learningVATT-MA-Medium pretrained on HowTo100M + AudioSet + YouTube8M · UCF101Top1: 88.28 -> 89.70Improved
add gradient realignment and gradient-based curriculum learningVATT-MA-Medium pretrained on HowTo100M + AudioSet + YouTube8M · HMDB51Top1: 65.84 -> 64.35Hurt
add gradient realignment and gradient-based curriculum learningVATT-MA-Medium pretrained on HowTo100M + AudioSet + YouTube8M · MSR-VTTRank: 56.00 -> 45.00Improved
add gradient realignment and gradient-based curriculum learningVATT-MA-Medium pretrained on HowTo100M + AudioSet + YouTube8M · ESC50Top1: 80.75 -> 84.50Improved
scale up the video-audio gradient and scale down video-text gradientVATT-MA-Medium · YouCook2Rank: 93.50 -> 168.90Hurt
apply cross-modality gradient realignmentVATT-MA-Medium pretrained on HowTo100M · YouCook2Rank: 93.50 -> 47.00Improved
apply gradient-based curriculum learningVATT-MA-Medium pretrained on HowTo100M · YouCook2Rank: 93.50 -> 40.00Improved
combine gradient realignment with curriculum learningVATT-MA-Medium pretrained on HowTo100M · YouCook2R@10: 17.10 -> 25.87Improved
use a progressively increasing gamma schedule from -0.3 to 0.0 (ablation)gradient-based curriculum learning on VATT · UCF101Top1: 10.05 -> 79.10Improved
raise the starting gamma from -0.3 to -0.2 (ablation)gradient-based curriculum learning on VATT · UCF101Top1: 79.10 -> 77.75Hurt
raise the starting gamma from -0.3 to -0.1 (ablation)gradient-based curriculum learning on VATT · UCF101Top1: 79.10 -> 44.49Hurt
use fixed gamma 0.0 instead of the progressive schedule (ablation)gradient-based curriculum learning on VATT · UCF101Top1: 79.10 -> 10.05Hurt

Earlier methods this paper compares with

Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.

Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy