Research map / Vision-language and multimodal / Cross-modal learning
Probing Inter-modality: Visual Parsing with Self-Attention for Vision-and-Language Pre-training
Hongwei Xue, Yupan Huang, Bei Liu, Houwen Peng, Jianlong Fu, Houqiang Li, Jiebo Luo
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace ResNet-101 grid features with Swin-T self-attention visual embedding (ablation) | VLP with 3-layer multi-modal Transformer · VQA dev (224px ablation) | accuracy: 65.44 -> 67.13 | Improved |
| replace ResNet-50 with Swin-T visual embedding (ablation) | VLP with 3-layer multi-modal Transformer · VQA dev (224px ablation) | accuracy: 64.65 -> 67.13 | Improved |
| replace ResNet-50 with Swin-T visual embedding (ablation) | VLP with 3-layer multi-modal Transformer · Inter-Modality Flow F^{1,3} | IMF: 0.437 -> 0.461 | Improved |
| add MFR with attention-based masking (ablation) | Swin-T VLP · VQA dev (224px ablation) | accuracy: 67.13 -> 67.74 | Improved |
| add MFR with attention-based masking (ablation) | Swin-T VLP · Inter-Modality Flow F^{1,3} | IMF: 0.461 -> 0.474 | Improved |
| use random masking instead of no MFR (ablation) | Swin-T VLP · VQA dev (224px ablation) | accuracy: 67.13 -> 67.24 | No clear effect |
| attention-based masking instead of random masking in MFR (ablation) | Swin-T VLP · VQA dev (224px ablation) | accuracy: 67.24 -> 67.74 | Improved |
| add CNN variant of MFR (cosine-distance correlation) (ablation) | ResNet-50 VLP · VQA dev (224px ablation) | accuracy: 64.65 -> 64.89 | Improved |
| add language-specific Transformer before cross-modal Transformer (ablation) | Swin-T + MFR VLP · VQA dev (224px ablation) | accuracy: 67.74 -> 67.96 | Improved |
| fully Transformer visual embedding (Swin) vs CNN in SOHO | end-to-end VLP · VQA test-dev | accuracy: 73.25 -> 74.00 | Improved |
| fully Transformer visual embedding (Swin) vs CNN in SOHO | end-to-end VLP · SNLI-VE val | accuracy: 85.00 -> 84.75 | No clear effect |
Closest papers by meaning
- MoDA: Modulation Adapter for Fine-Grained Visual Understanding in Instructional MLLMs (ICML 2026 · similarity 0.90)
- Bridging Vision and Language Spaces with Assignment Prediction (ICLR 2024 · similarity 0.89)
- Modality-Specialized Synergizers for Interleaved Vision-Language Generalists (ICLR 2025 · similarity 0.89)
- Probing Visual Language Priors in VLMs (ICML 2025 · similarity 0.89)
- VL-SAE: Interpreting and Enhancing Vision-Language Alignment with a Unified Concept Set (NeurIPS 2025 · similarity 0.89)
- MMRL: Multi-Modal Representation Learning for Vision-Language Models (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
