Research map / Vision-language and multimodal / CLIP and zero-shot
VLUE: A Multi-Task Multi-Dimension Benchmark for Evaluating Vision-Language Pre-training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Evaluate fine-tuned VLP models on newly annotated OOD test sets (MaRVL-image based) instead of in-domain COCO/VG test sets | VLP models (ALBEF, X-VLM, UNITER, etc.) · MSCOCO text retrieval, NLVR2, VQA, RefCOCO+ | R@1 / accuracy, in-domain vs OOD: R@1 drops from 80.4 to 64.1 (TR); 63.1 to 51.5 (IR); NLVR2 84.2 to 74.2; VQA 78.1 to 51.7 | Hurt |
| Image captioning evaluated with OFA vs encoder-only VLP models on COCO and OOD captions | OFA · COCO Captioning / OOD | CIDER / CIDER-OOD | Mixed |
| Use object-detection-based visual features (VL-T5) vs vision transformer encoders for captioning | VL-T5 vs X-VLM/OFA · COCO Captioning OOD | BLEU4-OOD: VL-T5 BLEU4-OOD 10.9 vs X-VLM 17.4 | Hurt |
| Use vision transformer image encoders (ALBEF, X-VLM, METER) instead of object-detection features | VLP models efficiency-performance trade-off · NLVR2 and VQA efficiency Pareto front | inference time vs performance | Helped |
| Compare VL-T5 vs ALBEF on NLVR2 and VQA efficiency | VL-T5 · NLVR2/VQA efficiency-performance | average inference time: requires 2.4 times longer inference than ALBEF while only marginally worse performance | Hurt |
| Correlation between in-domain and OOD performance across models | VLP models · VLUE in-domain vs OOD | Pearson correlation: rho = 0.75 | No clear effect |
Closest papers by meaning
- Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.92)
- VHELM: A Holistic Evaluation of Vision Language Models (NeurIPS 2024 · similarity 0.92)
- Probing Visual Language Priors in VLMs (ICML 2025 · similarity 0.92)
- Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.91)
- What matters when building vision-language models? (NeurIPS 2024 · similarity 0.91)
- Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation (ICLR 2026 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.