Research map / Large language models / Agents, code and math
MM-OPERA: Benchmarking Open-ended Association Reasoning for Large Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MM-OPERA, an 11,497-instance open-ended benchmark (Remote-Item Association and In-Context Association) for LVLM association reasoning, scored by LLM-as-a-Judge strategies including a process-reward judge (PR-Judge).
Problem
Existing benchmarks evaluate LVLM association only with closed-ended, option-based tasks, which may bias models and cannot assess long-form, multi-step open-ended association reasoning.
Approach
Re-develops the Remote Associates Test into two open-ended multimodal tasks requiring free-form answers with explicit association reasoning paths, scored by a holistic rubric plus a process-reward LLM judge that rates reasonableness, distinctiveness and knowledgeability per hop.
Key results
- On RIA, GPT-4o-based judging gives o4-mini 60.33% SR and 19.86% HR-4 versus human 61.88% SR and 22.84% HR-4; the best model is far below humans on ICA HR-4 (Gemini-2.5-Pro-Preview 12.85% vs human 31.65%).
- PR-Judge agreement with 8 domain experts: r = 0.72 for Reasonableness, r = 0.68 for Distinctiveness, 83.5% accuracy for Knowledgeability (Cohen's Kappa 0.65).
- Most models score higher on RIA than ICA, with ICA reasoning scores below 0.7 for all models while RIA scores exceed 1.1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use In-Context Association instead of Remote-Item Association | Most evaluated LVLMs · RIA vs ICA | Score Rate (SR) | Hurt |
| Use Gemini-1.5-Flash instead of Gemini-1.5-Pro | Gemini models · RIA | SR (%): 45.34 -> 55.86 | Helped |
| Use Gemini-1.5-Flash instead of Gemini-1.5-Pro | Gemini models · RIA; 500 random samples | Responses receiving 1-point scores: nearly 20% -> <10% | Helped |
| Increase association-path complexity from 1–2 hops to 4 hops | Evaluated LVLMs · RIA and ICA | Score rate (%): around 50–60% -> 29–53% | Hurt |
| Evaluate with PR-Judge rather than outcome-based regular scoring | Open-ended association responses · 100 reasoning paths | Reasoning quality distinction: 1.3 vs. 1.8 | Helped |
Surprising findings
- Gemini-1.5-Flash outperforms the larger Gemini-1.5-Pro on RIA (55.86% vs 45.34% SR), with Pro's conservative refusals driving many 1-point scores.
- Most models perform better on non-English RIA samples than English ones, which the authors attribute to precise cultural-fact retrieval.
Limitations
- Judge models (GPT-4o, DeepSeek-V3) may still introduce bias despite mixed-judge design and bias tests.
- Human baseline uses only 24 participants on a sampled subset of items.
- Non-English samples follow a long-tail distribution, limiting coverage of some languages.
Experimental setup
Datasets: RAT LI-RAT
Benchmarks: MM-OPERA
Metrics: Holistic score rate (SR) HR-3 HR-4 delta HR Reasoning score Reasonableness Distinctiveness Knowledgeability harmonic mean SR
Baselines: Claude-3.5-Sonnet Gemini-1.5-Flash Gemini-1.5-Pro Gemini-2.5-Pro-Preview o4-mini GPT-4o Qwen-VL-Max Qwen2.5-VL-7B-Instruct Yi-VL-34B Human
Builds on: Remote Associates Test process reward models The Labyrinth of Links
Closest papers by meaning
- FlexAC: Towards Flexible Control of Associative Reasoning in Multimodal Large Language Models (NeurIPS 2025 · similarity 0.89)
- What, Whether and How? Unveiling Process Reward Models for Thinking with Images Reasoning (AAAI 2026 · similarity 0.88)
- The Labyrinth of Links: Navigating the Associative Maze of Multi-modal LLMs (ICLR 2025 · similarity 0.88)
- Unveiling the Compositional Ability Gap in Vision-Language Reasoning Model (NeurIPS 2025 · similarity 0.88)
- Video-Only ToM: Enhancing Theory of Mind in Multimodal Large Language Models (CVPR 2026 · similarity 0.88)
- MARVEL: Multidimensional Abstraction and Reasoning through Visual Evaluation and Learning (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.