Research map / Vision-language and multimodal / Human and embodied modeling
Seeing Out of the Box: End-to-End Pre-Training for Vision-Language Representation Learning
Zhicheng Huang, Zhaoyang Zeng, Yupan Huang, Bei Liu, Dongmei Fu, Jianlong Fu
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use end-to-end SOHO with a trainable CNN and visual dictionary instead of region-based VLPT | SOHO · MSCOCO 5K test* (TR) | R@1: 64.4 -> 66.4 | Improved |
| use end-to-end SOHO with a trainable CNN and visual dictionary instead of region-based VLPT | SOHO · VQA2.0 test-std | VQA-score: 72.91 -> 73.47 | Improved |
| use end-to-end SOHO with a trainable CNN and visual dictionary instead of region-based VLPT | SOHO · NLVR² test-P | accuracy: 75.80 -> 77.32 | Improved |
| use end-to-end SOHO with a trainable CNN and visual dictionary instead of region-based VLPT | SOHO · SNLI-VE test | accuracy: 78.28 -> 84.95 | Improved |
| add the visual dictionary (VD) (ablation) | SOHO · MSCOCO 1K test (TR) | R@1: 72.80 -> 75.50 | Improved |
| add the visual dictionary (VD) (ablation) | SOHO · NLVR² test-P | accuracy: 62.61 -> 65.32 | Improved |
| set visual dictionary size to 1024 (ablation) | SOHO · MSCOCO 1K test (TR) | R@1: 72.80 -> 73.40 | Improved |
| set visual dictionary size to 2048 (ablation) | SOHO · MSCOCO 1K test (TR) | R@1: 72.80 -> 75.50 | Improved |
| set visual dictionary size to 4096 (ablation) | SOHO · MSCOCO 1K test (TR) | R@1: 72.80 -> 71.20 | Hurt |
| set visual dictionary size to 8192 (ablation) | SOHO · MSCOCO 1K test (TR) | R@1: 72.80 -> 72.10 | Hurt |
| replace the region-based BUTD inference pipeline with SOHO's whole-image CNN pipeline | SOHO · V100 inference experiment | inference time: 464ms -> 44ms | Improved |
Closest papers by meaning
- Probing Inter-modality: Visual Parsing with Self-Attention for Vision-and-Language Pre-training (NeurIPS 2021 · similarity 0.91)
- Unsupervised Vision-and-Language Pre-Training via Retrieval-Based Multi-Granular Alignment (CVPR 2022 · similarity 0.91)
- Locality Alignment Improves Vision-Language Models (ICLR 2025 · similarity 0.90)
- VL-SAE: Interpreting and Enhancing Vision-Language Alignment with a Unified Concept Set (NeurIPS 2025 · similarity 0.90)
- DH-Set: Improving Vision-Language Alignment with Diverse and Hybrid Set-Embeddings Learning (CVPR 2025 · similarity 0.89)
- Deep Pre-Alignment for VLMs (ICML 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
