Research map / Vision-language and multimodal / Multimodal LLMs

JourneyBench: A Challenging One-Stop Vision-Language Understanding Benchmark of Generated Images

NeurIPS 2024 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces JourneyBench, a ~13.5K-sample human-annotated benchmark of Midjourney-generated imaginary images spanning five VLU tasks (complementary MCOT, multi-image VQA, imaginary captioning, hallucination-trigger VQA, fine-grained retrieval with sample-specific distractors), showing even top models struggle.

Problem

Existing vision-language understanding benchmarks use common real-world images, so MLLMs can score well via language biases rather than genuine visual understanding; current MCOT, retrieval, and captioning benchmarks are saturated, redundant, or not fine-grained.

Approach

Builds VLU tasks on retrieved prompt-based generated (Midjourney) images that are unusual and fictional, makes MCOT strictly complementary so both modalities are required, adds sample-specific hard-negative distractors via an adversarial human-machine-in-the-loop process, and scales multi-image reasoning with generated images.

Key results

• GPT-4o reaches 62.18% accuracy on JourneyBench MCOT, while language-only GPT-4o gets 16.64%, and human performance is 84%.
• GPT-4o achieves 56.39% on multi-image VQA versus 73.42% on Mantis-Eval, and the best retrieval models drop to about 63.7% image R@1 with distractors (InternVL-G-14B).
• On HaloQuest-style hallucination triggers, LLaVA-NeXT-Qwen-110B scores 60.03% vs. 47.86% for GPT-4o.

Measured effects reported in the paper

Surprising findings

• LLaVA-NeXT-Qwen-110B outperforms GPT-4o and GPT-4V significantly on the hallucination-trigger VQA task.
• Adding GPT-4o-generated captions to MCOT questions does not significantly improve GPT-4o accuracy.

Limitations

• Retrieval distractors and evaluation rely on LLM-based answer extraction and verification (Llama-3), with only manual spot checks.
• Several models were evaluated on subsets of tasks, and some reported numbers are inconsistent across tables (e.g., mixed sizes and scores).

Experimental setup

Datasets: Midjourney generated images GSM8K COPA HaloQuest MS-COCO Flickr30K ScienceQA MathVista Mantis-Eval

Benchmarks: JourneyBench Flickr30K MS-COCO Mantis-Eval ScienceQA HaloQuest

Metrics: Recall@k BLEU ROUGE CIDEr METEOR accuracy joint consistency accuracy

Baselines: ALBEF CLIP X2-VLM BEiT3 BLIP-2 OpenCLIP-CoCa InternVL LLaVA-NeXT VILA Mantis MiniGPT-4 mPLUG-Owl

Builds on: HaloQuest GSM8K Mantis-Eval ScienceQA COPA

Closest papers by meaning

• MMFineReason: Closing the Multimodal Reasoning Gap via Open Data-Centric Methods (NeurIPS 2026 · similarity 0.91)
• MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI (ICML 2024 · similarity 0.91)
• LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models (ICLR 2026 · similarity 0.91)
• Perception Tokens Enhance Visual Reasoning in Multimodal Language Models (CVPR 2025 · similarity 0.91)
• Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.91)
• Don't Show Pixels, Show Cues: Unlocking Visual Tool Reasoning in Language Models via Perception Programs (CVPR 2026 · similarity 0.91)