Research map / Vision-language and multimodal / Multimodal LLMs

The Labyrinth of Links: Navigating the Associative Maze of Multi-modal LLMs

ICLR 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Unified any-modality language models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes an annotation-free method to convert existing labeled image datasets into multi-image association tasks and benchmarks zero-shot MLLMs and humans on single-step, synchronous, and asynchronous association over attributes, affordances, and actions.

Problem

Existing MLLM benchmarks do not evaluate association, the ability to link a new observation with prior practice memory over multiple steps; current MLLMs are assumed to lack this capability.

Approach

Build association chains from shared semantic labels (adjectives and verbs) in existing datasets (OCL, Pangea) without new annotation, then evaluate multi-step zero-shot association with three memory strategies (structured, natural language, chain memory) and an iterative deduction step.

Key results

• Gemini-1.5-Flash reaches a mean-step of 27.8 in attribute association versus 140.2 for humans, per the introduction.
• In attribute synchronous association, human experts reach a mean step of about 350 while open-source models stay below 20.
• Best single-step attribute success ratio is 89.87 for GPT-4o with NLM memory, below the human ceiling of 100.

Measured effects reported in the paper

Surprising findings

• Natural language memory outperformed structured memory and chain memory for most models, which the authors attribute to MLLMs being trained mostly on unstructured internet data.
• Under the deduction step, affordance deduction success ratios are very low for most models (e.g., 15.98 for Qwen2-VL), with GPT-4o at 27.30 average.

Limitations

• Evaluation is limited to zero-shot settings with tuning-free prompting; no fine-tuning on association data.
• Uses only three open-source MLLMs and two closed-source models; the MoE comparison covers a combination of only three open-source models.
• Benchmark relies on existing annotations from OCL and Pangea, and the memory strategies rely on hand-set hyperparameters (repetition weight 1.0, forgetting decrement 0.2).

Experimental setup

Datasets: Object Concept Learning (OCL) Pangea HMDB

Benchmarks: association benchmark (OCL attributes/affordances, Pangea actions)

Metrics: success ratio max step mean step

Baselines: GPT-4o Gemini-1.5-Flash GPT-4V LLaVA-OneVision Qwen2-VL mPLUG-Owl3 Human experts

Builds on: Object Concept Learning (OCL) Pangea chain-of-thought

Closest papers by meaning

• Can Large Language Models Help Multimodal Language Analysis? MMLA: A Comprehensive Benchmark (NeurIPS 2025 · similarity 0.89)
• On Path to Multimodal Generalist: General-Level and General-Bench (ICML 2025 · similarity 0.89)
• SEED-Bench: Benchmarking Multimodal Large Language Models (CVPR 2024 · similarity 0.88)
• MM-OPERA: Benchmarking Open-ended Association Reasoning for Large Vision-Language Models (NeurIPS 2025 · similarity 0.88)
• FlexAC: Towards Flexible Control of Associative Reasoning in Multimodal Large Language Models (NeurIPS 2025 · similarity 0.88)
• MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark (ICML 2024 · similarity 0.88)