Research map / Vision-language and multimodal / Multimodal LLMs

Test-Time Matching: Unlocking Compositional Reasoning in Multimodal Models

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that the Winoground-style group score underestimates multimodal models on compositional benchmarks, proposes a matching-based group score and an iterative test-time self-training method (TTM) using confident matching pseudo-labels.

Problem

Compositional reasoning benchmarks (Winoground, MMVP-VLM, ColorSwap, SugarCrepe, WhatsUp) report multimodal models at or below chance under the standard group score, which may be an evaluation artifact.

Approach

Replace the isolated pairwise group score with a bijective-matching score that asks whether the ground-truth matching has the highest total similarity; then iteratively self-train on confident matching-induced pseudo-labels with a decaying threshold at test time.

Key results

• GPT-4.1 under the matching score (Simple Match) reaches 91.38 on Winoground, surpassing the estimated human 85.5, up from 69.75 raw.
• TTM lifts SigLIP-B16 to 89.44 on MMVP-VLM, beating GPT-4.1 (88.52) and setting a new state of the art.
• TTM gives up to 85.7% relative gain on WhatsUp A (1x4) with CLIP-B32 (30.58 to 56.8).

Surprising findings

• Under random guessing, the group score for 2x2 groups is 1/6 while the matching score is 1/2, so the standard metric is substantially pessimistic.
• An ascending threshold schedule for pseudo-labels yields no gains because the model overfits to false positives in the first iteration.

Limitations

• Gains depend on pseudo-label precision and hand-tuned initial thresholds per dataset-model pair.
• Simple Match and TTM results on MMVP-VLM and Winoground rely on small benchmark sizes with high variance across runs.
• Generative MLLM experiments only use SmolVLM-256M, and the benchmarks are designed for contrastive models.

Experimental setup

Datasets: Winoground MMVP-VLM ColorSwap SugarCrepe WhatsUp

Benchmarks: Winoground MMVP-VLM ColorSwap SugarCrepe WhatsUp

Metrics: group score matching score accuracy relative gain relative error reduction

Baselines: GPT-4.1 (raw group score) GPT-4V with prompt tuning GPT-4o multi-agent system CLIP-B16 SigLIP-L16 SmolVLM-256M-Instruct

Builds on: Winoground group score VQAScore SigLIP CLIP test-time training

Closest papers by meaning

• LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models (ICLR 2026 · similarity 0.91)
• Unveiling the Compositional Ability Gap in Vision-Language Reasoning Model (NeurIPS 2025 · similarity 0.90)
• ConMe: Rethinking Evaluation of Compositional Reasoning for Modern VLMs (NeurIPS 2024 · similarity 0.89)
• AStar: Boosting Multimodal Reasoning with Automated Structured Thinking (AAAI 2026 · similarity 0.89)
• RPTS: Tree-Structured Reasoning Process Scoring for Faithful Multimodal Evaluation (AAAI 2026 · similarity 0.89)
• ThinkMorph: Emergent Properties in Multimodal Interleaved Chain-of-Thought Reasoning (ICLR 2026 · similarity 0.89)