Research map / Vision-language and multimodal / Human and embodied modeling
Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Models
Jize Cao, Zhe Gan, Yu Cheng, Licheng Yu, Yen-Chun Chen, Jingjing Liu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| go deeper from layer 0 to layer 11 in single-stream model | UNITER-base · Flickr30k Entities | NMI: 0.36 -> 0.20 | Improved |
| go deeper from layer 0 to layer 3 in two-stream cross-modal encoder | LXMERT · Flickr30k Entities | NMI: 0.42 -> 0.75 | Hurt |
| use attention probers on layer 5 embedding instead of 144 attention heads for coreference classification | UNITER-base · Flickr30k coreference VCC | accuracy (%): 75.10 -> 94.05 | Improved |
| use attention probers on layer 1 embedding instead of 144 attention heads for coreference classification | LXMERT · Flickr30k coreference VCC | accuracy (%): 54.47 -> 93.35 | Improved |
| probe coreference detection from attention heads vs random guess | UNITER-base · Flickr30k coreference VCD | accuracy (%): 50.00 -> 52.04 | No clear effect |
| train the model instead of random weights, visual relation classification | UNITER-base · Visual Genome VRC | accuracy (%): 11.4 -> 56.64 | Improved |
| pair image with mismatched caption instead of matched, visual relation classification layer 10 (ablation) | LXMERT · Visual Genome VRC | accuracy (%): 69.66 -> 12.07 | Hurt |
| pair image with mismatched caption instead of matched, visual relation classification layer 12 (ablation) | UNITER-base · Visual Genome VRI | accuracy (%): 86.07 -> 72.96 | Hurt |
| add multimodal pre-training on top of BERT initialization: compare UNITER with BERT on SentLen | UNITER-base · SentEval SentLen | accuracy (%): 96.2 -> 88.8 | Hurt |
| initialize language Transformer from BERT-base (UNITER) instead of not (LXMERT) | UNITER-base · SentEval BShift | accuracy (%): 64.9 -> 81.6 | Improved |
| use single-stream instead of two-stream on CoordInv | UNITER-base · SentEval CoordInv | accuracy (%): 58.7 -> 62.1 | Improved |
| train the model instead of random weights, visual coreference classification | UNITER-base · Flickr30k VCC | accuracy (%): 58.0 -> 75.10 | Improved |
Closest papers by meaning
- Unveiling Visual Perception in Language Models: An Attention Head Analysis Approach (CVPR 2025 · similarity 0.92)
- Seeing is Understanding: Unlocking Causal Attention into Modality-Mutual Attention for Multimodal LLMs (ICML 2026 · similarity 0.91)
- Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training (ICLR 2026 · similarity 0.91)
- Understanding Language Prior of LVLMs by Contrasting Chain-of-Embedding (ICLR 2026 · similarity 0.91)
- Where Does Vision Meet Language? Understanding and Refining Visual Fusion in MLLMs via Contrastive Attention (CVPR 2026 · similarity 0.91)
- What's in the Image? A Deep-Dive into the Vision of Vision Language Models (CVPR 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
