Research map / Vision-language and multimodal / Multimodal LLMs

Activating Visual Context and Commonsense Reasoning Through Masked Prediction in VLMs

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Reinforcement learning for visual reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces the MPCC masked-prediction task and MPCC-Eval benchmark (1,114 COCO-based masked-image questions), and compares prompt, SFT, GRPO-RFT and a new RFT-with-prior-sampling method on Qwen2.5-VL-3B.

Problem

VLMs struggle to integrate visual context and commonsense knowledge for reasoning; existing RL-for-VLM work is confined to perception tasks or reduces images to text. The paper asks how to activate and evaluate such reasoning.

Approach

Mask key visual entities in COCO images and train/evaluate VLMs to infer the masked object from context and commonsense; propose RFT with prior sampling, which replaces one GRPO policy sample per group with an annotated reasoning trace so that scarce reasoning labels guide the policy.

Key results

• On MPCC-Eval, GPT-4o reaches 71.30 (Cho.E) and 58.36 (Cho.H) average accuracy, versus 59.09 and 30.15 for the Qwen2.5-VL-3B base.
• RFT with prior sampling raises Qwen2.5-VL-3B hard easy-choice accuracy on in-distribution data from 59.09 to 72.59 and gives the best OOD easy-choice score (74.20) alongside RFT.
• On BLINK cross-task transfer, RFT with prior sampling improves Semantic Correspondence by +10.40 points (29.89 to 40.29), while Visual-RFT drops 14.51 points on Relative Depth.

Surprising findings

• Visual-RFT and PR1 fine-tuning on perception tasks degrade performance on several BLINK reasoning tasks.
• SFT gives the best in-distribution hard-choice score but weaker OOD generalization than RFT with prior sampling.

Limitations

• Training data is limited and the gap to natural-language reasoning remains.
• Relies on small-scale experiments primarily on a single 3B base model.
• Prior-sampling results are mixed: in-distribution performance trails SFT+RFT.

Experimental setup

Datasets: COCO BLINK

Benchmarks: MPCC-Eval BLINK

Metrics: accuracy

Baselines: Visual-RFT PR1 Reason-RFT GPT-4o GPT-4o-mini Qwen2.5-VL-7B InternVL3-8B LLaVA-1.6

Builds on: GRPO DeepSeek-R1 Visual-RFT Reason-RFT PR1 OpenR1

Closest papers by meaning

• Reason-RFT: Reinforcement Fine-Tuning for Visual Reasoning of Vision Language Models (NeurIPS 2025 · similarity 0.92)
• More Thought, Less Accuracy? On the Dual Nature of Reasoning in Vision-Language Models (ICLR 2026 · similarity 0.92)
• Unveiling the Compositional Ability Gap in Vision-Language Reasoning Model (NeurIPS 2025 · similarity 0.91)
• On Robustness and Chain-of-Thought Consistency of RL-Finetuned VLMs (ICML 2026 · similarity 0.91)
• Multi-step Visual Reasoning with Visual Tokens Scaling and Verification (NeurIPS 2025 · similarity 0.91)
• ViCrit: A Verifiable Reinforcement Learning Proxy Task for Visual Perception in VLMs (NeurIPS 2025 · similarity 0.91)