AI Paper Atlas

Research map / Vision-language and multimodal / Cross-modal learning

Probing Inter-modality: Visual Parsing with Self-Attention for Vision-and-Language Pre-training

Hongwei Xue, Yupan Huang, Bei Liu, Houwen Peng, Jianlong Fu, Houqiang Li, Jiebo Luo

NeurIPS 2021 · Vision-language and multimodal › Cross-modal learning · approach: Multi-modality image fusion · 32 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
replace ResNet-101 grid features with Swin-T self-attention visual embedding (ablation)VLP with 3-layer multi-modal Transformer · VQA dev (224px ablation)accuracy: 65.44 -> 67.13Improved
replace ResNet-50 with Swin-T visual embedding (ablation)VLP with 3-layer multi-modal Transformer · VQA dev (224px ablation)accuracy: 64.65 -> 67.13Improved
replace ResNet-50 with Swin-T visual embedding (ablation)VLP with 3-layer multi-modal Transformer · Inter-Modality Flow F^{1,3}IMF: 0.437 -> 0.461Improved
add MFR with attention-based masking (ablation)Swin-T VLP · VQA dev (224px ablation)accuracy: 67.13 -> 67.74Improved
add MFR with attention-based masking (ablation)Swin-T VLP · Inter-Modality Flow F^{1,3}IMF: 0.461 -> 0.474Improved
use random masking instead of no MFR (ablation)Swin-T VLP · VQA dev (224px ablation)accuracy: 67.13 -> 67.24No clear effect
attention-based masking instead of random masking in MFR (ablation)Swin-T VLP · VQA dev (224px ablation)accuracy: 67.24 -> 67.74Improved
add CNN variant of MFR (cosine-distance correlation) (ablation)ResNet-50 VLP · VQA dev (224px ablation)accuracy: 64.65 -> 64.89Improved
add language-specific Transformer before cross-modal Transformer (ablation)Swin-T + MFR VLP · VQA dev (224px ablation)accuracy: 67.74 -> 67.96Improved
fully Transformer visual embedding (Swin) vs CNN in SOHOend-to-end VLP · VQA test-devaccuracy: 73.25 -> 74.00Improved
fully Transformer visual embedding (Swin) vs CNN in SOHOend-to-end VLP · SNLI-VE valaccuracy: 85.00 -> 84.75No clear effect

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy