Research map / Vision-language and multimodal / Multimodal LLMs

StemBind: When MLLMs Get Lost Between Rules and Instances in Abstract Visual Reasoning

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces StemBind, a shared-stem AVR benchmark with P/R/F probes and S1–S4 process annotations, and evaluates 24 MLLM configurations to show a rule-to-instance binding gap.

Problem

MLLMs often identify the rule in abstract visual reasoning yet choose the wrong answer, but existing AVR benchmarks score only final answers and cannot localize where the failure occurs.

Approach

Probe the same visual stem with perception, rule, and full-item questions plus Sternberg-based S1–S4 stage annotations, and use stage-wise stimulus augmentation (SSA) to test which stage's information repairs failures, allowing within-stem attribution of errors.

Key results

• Rule accuracy exceeds full-item accuracy on 22 of 24 direct-mode configurations, often by 20-34 points.
• Even when perception and rule are both correct on the same stem, models answer full items incorrectly 51.2% of the time (Binding Gap 0.51).
• Qwen3.5 S3 (Map) is the weakest L2 stage on every full-split row (mean StepAcc 30.5%); thinking mode lowers rule and full-item accuracy on all paired rows.

Surprising findings

• Thinking mode raises perception accuracy but lowers both rule and full-item accuracy across paired comparisons.
• Scaling within families does not reliably close the full-item gap; Qwen3.5-122B-A10B collapses on rule probes relative to 27B.

Limitations

• Benchmark scale is limited to 2,298 stems and English-only prompts.
• Stage and SSA localization rely on a GPT-4o trace judge and are primarily done on Qwen3.5; S3 labels are noisiest.
• Findings are behavioral diagnostics rather than mechanistic claims about internal binding.

Experimental setup

Datasets: StemBind

Benchmarks: StemBind

Metrics: accuracy P/R/F accuracy Binding Gap StepAcc SSA gain R-F gap

Baselines: MARVEL PuzzleVQA VisualPuzzles VisuLogic VisuRiddles VRIQ VisRes Bench VISFACTOR M3GIA SpatialViz-Bench

Builds on: Sternberg componential theory of induction CHC theory

Closest papers by meaning

• VisuRiddles: Fine-grained Perception is a Primary Bottleneck for Multimodal Large Language Models in Abstract Visual Reasoning (ICLR 2026 · similarity 0.92)
• Unveiling Fine-Grained Visual Traces: Evaluating MultiModal Interleaved Reasoning Chains in Multimodal STEM Tasks (NeurIPS 2026 · similarity 0.90)
• Look-Back: Implicit Visual Re-focusing in MLLM Reasoning (AAAI 2026 · similarity 0.90)
• PRISM-Bench: A Benchmark of Puzzle-Based Visual Tasks with CoT Error Detection (NeurIPS 2026 · similarity 0.90)
• VisRes Bench: On Evaluating the Visual Reasoning Capabilities of VLMs (CVPR 2026 · similarity 0.90)
• CodePercept: Code-Grounded Visual STEM Perception for MLLMs (CVPR 2026 · similarity 0.90)