Research map / Vision-language and multimodal / Multimodal LLMs

Visual Para-Thinker: Divide-and-Conquer Reasoning for Visual Comprehension

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Visual Para-Thinker, a parallel divide-and-conquer reasoning framework for MLLMs that generates visually partitioned reasoning paths isolated by Pa-Attention and distinguished by LPRoPE, fine-tuned from Qwen2.5-VL-3B/7B and run on vLLM.

Problem

Test-time scaling via sequential long chain-of-thought plateaus in exploration, and parallel thinking has not been extended to multimodal visual reasoning.

Approach

Visual partitioning (block-based and scan-order) turns the image into parallel reasoning paths; Pa-Attention isolates paths, LPRoPE adds learnable path embeddings before RoPE to keep paths distinguishable yet position-unbiased, and a summary stage merges paths; KV-cache reuse in vLLM makes the parallel decoding efficient.

Key results

• Visual Para-Thinker-7B reaches 74.0 average on counting/perception/hallucination vs 67.2 for Qwen2.5-VL-7B, with V* at 82.2 vs 75.9.
• Visual Para-Thinker-3B reaches 88.5% average Acc@0.5 on RefCOCO vs 84.8 for Qwen2.5-VL-3B.
• KV cache reuse in the vLLM implementation raises throughput from 104 to 123 token/s versus no reuse and cuts V* inference time from 364s to 312s.

Surprising findings

• Block-based partitioning beats scan-order on fine-grained tasks while scan-order wins on counting, motivating a hybrid training scheme.
• Majority voting@4 and the Sequential SFT baseline barely improve over the base Qwen2.5-VL-7B on most benchmarks.

Limitations

• Parallel paths are generated from fixed visual partitions (block or scan order), limiting path diversity to predefined patterns.
• Training relies on a large teacher model (Qwen3-VL-235B) to produce the 163K reasoning paths.
• Comparison to RL-trained reasoning is absent; the authors note the link between parallel reasoning and RL is left for future work.

Experimental setup

Datasets: LVIS LAION Microsoft COCO PixMo-Count RefCOCO RefCOCO+ RefCOCOg

Benchmarks: CountBench PixMo-Count V* MMVP HallusionBench RefCOCO RefCOCO+ RefCOCOg

Metrics: accuracy Acc@0.5 time (s) throughput (token/s)

Baselines: Qwen2.5-VL Sequential (SFT CoT) GRPO Majority voting@4 GPT-4o Gemini2.5-Pro

Builds on: Para-Thinker Multiverse RoPE vLLM Qwen2.5-VL

Closest papers by meaning

• VGR: Visual Grounded Reasoning (ICLR 2026 · similarity 0.90)
• Look-Back: Implicit Visual Re-focusing in MLLM Reasoning (AAAI 2026 · similarity 0.90)
• SPARC: Separating Perception And Reasoning Circuits for Test-time Scaling of VLMs (ICML 2026 · similarity 0.90)
• Multi-step Visual Reasoning with Visual Tokens Scaling and Verification (NeurIPS 2025 · similarity 0.89)
• Imagine While Reasoning in Space: Multimodal Visualization-of-Thought (ICML 2025 · similarity 0.89)
• Revisiting the Necessity of Lengthy Chain-of-Thought in Vision-centric Reasoning Generalization (CVPR 2026 · similarity 0.89)