AI Paper Atlas

Research map / Vision-language and multimodal / Human and embodied modeling

Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Models

Jize Cao, Zhe Gan, Yu Cheng, Licheng Yu, Yen-Chun Chen, Jingjing Liu

ECCV 2020 · Vision-language and multimodal › Human and embodied modeling · approach: Vision-language pretraining · 112 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
go deeper from layer 0 to layer 11 in single-stream modelUNITER-base · Flickr30k EntitiesNMI: 0.36 -> 0.20Improved
go deeper from layer 0 to layer 3 in two-stream cross-modal encoderLXMERT · Flickr30k EntitiesNMI: 0.42 -> 0.75Hurt
use attention probers on layer 5 embedding instead of 144 attention heads for coreference classificationUNITER-base · Flickr30k coreference VCCaccuracy (%): 75.10 -> 94.05Improved
use attention probers on layer 1 embedding instead of 144 attention heads for coreference classificationLXMERT · Flickr30k coreference VCCaccuracy (%): 54.47 -> 93.35Improved
probe coreference detection from attention heads vs random guessUNITER-base · Flickr30k coreference VCDaccuracy (%): 50.00 -> 52.04No clear effect
train the model instead of random weights, visual relation classificationUNITER-base · Visual Genome VRCaccuracy (%): 11.4 -> 56.64Improved
pair image with mismatched caption instead of matched, visual relation classification layer 10 (ablation)LXMERT · Visual Genome VRCaccuracy (%): 69.66 -> 12.07Hurt
pair image with mismatched caption instead of matched, visual relation classification layer 12 (ablation)UNITER-base · Visual Genome VRIaccuracy (%): 86.07 -> 72.96Hurt
add multimodal pre-training on top of BERT initialization: compare UNITER with BERT on SentLenUNITER-base · SentEval SentLenaccuracy (%): 96.2 -> 88.8Hurt
initialize language Transformer from BERT-base (UNITER) instead of not (LXMERT)UNITER-base · SentEval BShiftaccuracy (%): 64.9 -> 81.6Improved
use single-stream instead of two-stream on CoordInvUNITER-base · SentEval CoordInvaccuracy (%): 58.7 -> 62.1Improved
train the model instead of random weights, visual coreference classificationUNITER-base · Flickr30k VCCaccuracy (%): 58.0 -> 75.10Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy