Research map / Vision-language and multimodal / Multimodal LLMs

Beyond Single Images: A Comprehensive Benchmark for Album-Level Vision-Language Understanding

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces AlbumBench, a benchmark of 27,051 images in 641 albums (from CUFED) with four album-organization tasks (intent selection, intent rating, group labeling, group clustering), and evaluates open and proprietary VLMs under visual and caption-based context.

Problem

Vision-language models are evaluated mainly on single images or short multi-image sets, not on album-level organization tasks that require user-intent understanding and joint context across dozens of non-temporally-consistent photos.

Approach

Defines album organization as intent-driven photo selection, rating, and grouping, and builds a benchmark from CUFED albums with five annotations per image, comparing visual-context inputs against caption-only (language-context) inputs.

Key results

• Qwen3-VL-235B-A22B (instruct) reaches F1 0.708 on intent selection with visual context, while thinking variants improve grouping; Qwen3-VL-32B-Thinking leads Group Clustering on all four metrics.
• Closed-source models generally outperform open-source instruct models, with GPT-5 (full thinking) achieving the best Group Labeling ARI of 0.661 among visual-context runs.
• Caption-only Gemini baselines perform comparably to visual-context runs, and tasks with language context show weak Spearman correlation with MMMU-val.

Surprising findings

• Caption-only baselines perform comparably to visual-context inputs, suggesting VLMs under-use image tokens for album tasks.
• Instruct models often violate the one-group-per-image constraint in grouping tasks, with failure rates up to 82% for InternVL3.5-8B.

Limitations

• Thinking-mode gains come with high computational and time cost.
• Open-source instruct models often fail to produce the required structured output without extensive prompt engineering.
• Albums are limited to 30-100 images from CUFED event types, which may not represent full personal photo galleries.

Experimental setup

Datasets: CUFED

Benchmarks: AlbumBench

Metrics: F1 precision recall mAP accuracy MAE RMSE ARI NMI Jaccard index

Baselines: GPT-5 Gemini-2.5-Pro Qwen3-VL InternVL3.5 Keye-VL-1.5 Gemini caption baseline

Builds on: CUFED MMMU MileBench MuirBench

Closest papers by meaning

• Image2Struct: Benchmarking Structure Extraction for Vision-Language Models (NeurIPS 2024 · similarity 0.87)
• JourneyBench: A Challenging One-Stop Vision-Language Understanding Benchmark of Generated Images (NeurIPS 2024 · similarity 0.86)
• The Photographer's Eye: Teaching Multimodal Large Language Models to See, and Critique Like Photographers (CVPR 2025 · similarity 0.86)
• What's in the Image? A Deep-Dive into the Vision of Vision Language Models (CVPR 2025 · similarity 0.85)
• Can Machines Understand Composition? Dataset and Benchmark for Photographic Image Composition Embedding and Understanding (CVPR 2025 · similarity 0.85)
• MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Models (ICLR 2025 · similarity 0.85)