Research map / Vision-language and multimodal / Multimodal LLMs

LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces LENS, a multi-level benchmark of 3.4K social-media images and 60K+ human-authored questions across perception, understanding, and reasoning tiers, plus SMEC, a prompt-only multi-expert self-collaboration framework for MLLMs.

Problem

Existing MLLM benchmarks are task-oriented and built on different data distributions, so they cannot measure synergistic effects of low-level perception on high-level reasoning, and they often use stale images that risk pretraining contamination.

Approach

A benchmark where all eight tasks share the same contemporary images with dense annotations, enabling controlled cross-task analysis; and SMEC, which has a single MLLM generate role-specific expert prompts, collect their answers, and integrate them into a consensus answer without training or external tools.

Key results

• On LENS, no evaluated MLLM exceeds 60% accuracy on reasoning tasks; Gemini2.5-Pro reaches 59.31% on SKI and QVQ-Max 58.86%.
• SMEC raises Qwen2.5VL-7B on the SKI subset from 39.80% (direct) to 43.24% with 3 iterations, above Self-Refine (40.51%) and majority voting (40.66%).
• SMEC improves Qwen2.5VL-32B on the full SKI test set from 51.54% to 54.66% (+3.12).

Surprising findings

• OC correlates more strongly with OCR (0.77) than with OE (0.46), contrary to intuition, which the authors attribute to shared fine-grained localization demands.
• Reasoning model QVQ-Max approaches closed-source performance on SKI despite lacking OD and VG capabilities.

Limitations

• Reasoning gains from SMEC are modest (about +3 points) and evaluated on a single task (SKI) with only Qwen2.5-VL models.
• Dataset bias from social-media platform demographics and uneven scenario coverage, as acknowledged in the ethics statement.
• Open-ended answers are scored by an LLM judge (GLM4-flash) without reported human validation of judge agreement.

Experimental setup

Datasets: LENS

Benchmarks: LENS

Metrics: accuracy AP50 ACC@0.5

Baselines: GPT-4o Gemini2.5-Pro Qwen2.5-VL InternVL3 Deepseek-VL2 Gemma3 QVQ-Max Kimi-VL-thinking Self-Refine Majority voting

Builds on: chain-of-thought prompting Self-Refine majority voting MetaGPT AgentVerse

Closest papers by meaning

• MMR-Life: Piecing Together Real-life Scenes for Multimodal Multi-image Reasoning (ICLR 2026 · similarity 0.94)
• RPTS: Tree-Structured Reasoning Process Scoring for Faithful Multimodal Evaluation (AAAI 2026 · similarity 0.93)
• Compose and Fuse: Revisiting the Foundational Bottlenecks in Multimodal Reasoning (ICLR 2026 · similarity 0.92)
• Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmark (ICML 2025 · similarity 0.92)
• VOILA: Evaluation of MLLMs For Perceptual Understanding and Analogical Reasoning (ICLR 2025 · similarity 0.92)
• AStar: Boosting Multimodal Reasoning with Automated Structured Thinking (AAAI 2026 · similarity 0.92)