Research map / Vision-language and multimodal / Multimodal LLMs

Mixture-of-Visual-Thoughts: Exploring Context-Adaptive Reasoning Mode Selection for General Visual Reasoning

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MoVT and AdaVaR, a two-stage (SFT cold-start then RL with AdaGRPO) framework that unifies text-based and visually-grounded reasoning modes in one LVLM and learns context-adaptive mode selection.

Problem

Visual reasoning methods specialize in one reasoning mode (text-based or visually-grounded), so each excels in some domains (math vs. object-centric tasks) but fails to generalize; a single model should integrate both and choose the right mode per input.

Approach

Unify text and grounded reasoning under mode-specific prefix tokens learned via SFT, then use AdaGRPO in RL: prefix-guided uniform sampling across modes, plus a mode-relative advantage (Gaussian CDF of mode reward distributions) assigned to the prefix token so the model learns which mode to select.

Key results

• AdaVaR-7B reaches 55.82 average accuracy across 8 benchmarks, above GPT-4o (53.20) and Qwen2.5-VL-7B (50.90), and best in its 7B group.
• AdaVaR-3B averages 50.84, best in the 3B group and close to Qwen2.5-VL-7B (50.90), and is the only 3B model outperforming Qwen2.5-VL-3B on all datasets.
• Removing both Ada-Adv and PG-Exp (equivalent to vanilla GRPO) drops AdaVaR-3B average from 50.8 to 49.6; single-mode baselines reach 48.5-48.8.

Surprising findings

• Two-stage SFT without RL (SFT-Adaptive-3B) produces a strong unwarranted preference for the grounded mode and underperforms the Stage 1 SFT model on math.
• Simple dataset-level mode ensembles, even with oracle information (Max(GRD,TXT)), underperform sample-level adaptive selection learned by AdaVaR.

Limitations

• Only two reasoning modes (text-based and bounding-box grounded) are integrated; extension to more modes is left open.
• Evaluation relies on a mode-switching fallback when outputs hit the token limit, and baselines were reproduced or evaluated by the authors.
• Models focus on English scenarios.

Experimental setup

Datasets: VoCoT R1-OneVision llava-CoT NuminaMath1.5 Virgo Geo170K OmniCount MM-Eureka LLaVA-OneVision SFT data InternVL SFT data

Benchmarks: MathVista MathVision MathVerse WeMath MMStar V* POPE SpatialScore

Metrics: accuracy average accuracy strict accuracy

Baselines: GPT-4o LLaVA-OV-7B InternVL3-8B Qwen2.5-VL-3B Qwen2.5-VL-7B LMM-R1 VLAA-Thinker GRIT ViGoRL MM-Eureka OVR-7B Orsta-7B

Builds on: GRPO DeepSeek-R1 Qwen2.5-VL VoCoT Orsta

Closest papers by meaning

• AStar: Boosting Multimodal Reasoning with Automated Structured Thinking (AAAI 2026 · similarity 0.88)
• VGR: Visual Grounded Reasoning (ICLR 2026 · similarity 0.88)
• SoTA with Less: MCTS-Guided Sample Selection for Data-Efficient Visual Reasoning Self-Improvement (NeurIPS 2025 · similarity 0.88)
• Multi-step Visual Reasoning with Visual Tokens Scaling and Verification (NeurIPS 2025 · similarity 0.87)
• VisuRiddles: Fine-grained Perception is a Primary Bottleneck for Multimodal Large Language Models in Abstract Visual Reasoning (ICLR 2026 · similarity 0.87)
• Reason-RFT: Reinforcement Fine-Tuning for Visual Reasoning of Vision Language Models (NeurIPS 2025 · similarity 0.87)