Research map / Vision-language and multimodal / Multimodal LLMs
Trust but Verify: Programmatic VLM Evaluation in the Wild
Viraj Prabhu, Senthil Purushwalkam, An Yan, Caiming Xiong, Ran Xu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| larger InternVL2 model size 2B to 26B | InternVL2 · PROVE | hscore: 73.96 -> 74.63 | Improved |
| larger InternVL2 model size 2B to 26B | InternVL2 · PROVE | tscore: 79.51 -> 79.23 | Hurt |
| scale LLaVA-1.5 from 7B to 13B | LLaVA-1.5 · PROVE | tscore: 82.58 -> 82.40 | No clear effect |
| scale LLaVA-1.5 from 7B to 13B | LLaVA-1.5 · PROVE | hscore: 72.67 -> 72.46 | No clear effect |
| newer LLaVA-Next-7B instead of LLaVA-1.5-7B | LLaVA · PROVE | hscore: 72.67 -> 74.28 | Improved |
| newer LLaVA-Next-7B instead of LLaVA-1.5-7B | LLaVA · PROVE | tscore: 82.58 -> 80.03 | Hurt |
| give the model the ground-truth caption (oracle) versus GPT-4o | LLaMA-3.1-8B oracle · PROVE | average: 78.72 -> 84.22 | Improved |
| GPT-4o versus Claude-3.5-Sonnet | closed-source VLMs · PROVE | tscore: 77.31 -> 80.92 | Improved |
| InternVL2-26B versus simpler LLaVA-1.5-7B on truthfulness | VLMs · PROVE | tscore: 82.58 -> 79.23 | Hurt |
| correlation between helpfulness and truthfulness across models | VLMs · PROVE | Pearson correlation: 0.03 | No clear effect |
| compare automatic metrics to human ratings | PROVE metrics · PROVE human study | Pearson correlation: 0.81 vs 0.45 | Mixed |
Closest papers by meaning
- MoHoBench: Assessing Honesty of Multimodal Large Language Models via Unanswerable Visual Questions (AAAI 2026 · similarity 0.89)
- NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samples (NeurIPS 2024 · similarity 0.88)
- Seeing but Not Believing: Probing the Disconnect Between Visual Attention and Answer Correctness in VLMs (ICLR 2026 · similarity 0.88)
- Prism: A Framework for Decoupling and Assessing the Capabilities of VLMs (NeurIPS 2024 · similarity 0.87)
- UPME: An Unsupervised Peer Review Framework for Multimodal Large Language Model Evaluation (CVPR 2025 · similarity 0.87)
- Improving Automatic VQA Evaluation Using Large Language Models (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
