Research map / Vision-language and multimodal / Multimodal LLMs

Token-Sparse Medical Multimodal Reasoning via Dual-Stream Reinforcement Learning

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Attention-based visual token pruning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ViToS, a dual-stream GRPO-based RL framework that trains one VLM to both predict a foreground bounding box and reason over grounding-pruned (and fused) visual tokens, with cross-feedback rewards and sequential optimization, for token-sparse medical VQA.

Problem

Medical VLMs struggle because medical images contain sparse decisive visual evidence amid redundant tokens, and existing visual token pruning methods are heuristic and not optimized end-to-end for reasoning; a unified RL framework combining grounding-aware token pruning and medical reasoning has not

Approach

A single policy VLM is trained with two cascaded branches: a localization branch predicting a bounding box, and a token-sparse reasoning branch that prunes/fuses visual tokens outside the box. Cross-feedback rewards pass each branch's accuracy to the other, and training proceeds sequentially (localization first, then reasoning with localization frozen) to avoid gradient conflict.

Key results

• On Lingshu-7B, ViToS averages 68.95 versus 63.68 for the instruct baseline (108.27% relative) while keeping 77% of visual tokens.
• On HuatuoGPT-Vision-7B, ViToS averages 59.17 versus 56.81 baseline (104.16%); on Lingshu-32B it reaches 70.21 vs 66.46 (105.64%).
• Inference time on Lingshu-7B is 11 minutes versus 47–54 minutes for pruning baselines (about 4.5x speedup, with vLLM).

Surprising findings

• Pruning-only GTP without token fusion drops performance by 5.4% on PMC-VQA and 4.3% on SLAKE for Lingshu-7B, while fusion is needed for stability.
• SFT failed to converge for the joint localization and reasoning format, motivating the RL choice.

Limitations

• Not designed for extreme-resolution medical images, where localization action space grows large.
• Relies on standard-resolution inputs and a bounding-box-annotated training set derived from object detection datasets.

Experimental setup

Datasets: ViTAR training dataset (18,222 VQA with boxes) PathVQA SLAKE VQA-RAD OmniMedVQA PMC-VQA MMMU-Med MedXpertQA

Benchmarks: PMC-VQA PathVQA SLAKE VQA-RAD OmniMedVQA MMMU-Med MedXpertQA

Metrics: accuracy average accuracy relative performance percentage inference time

Baselines: VisionZip VScan FastV PDrop Med-R1 MedVLM-R1 MedCCO MedVLThinker ViTAR DeepEyes PixelReasoner

Builds on: GRPO Vision-SR1 ViTAR DeepSeek-R1

Closest papers by meaning

• MedReasoner: Reinforcement Learning Drives Reasoning Grounding from Clinical Thought to Pixel-Level Precision (AAAI 2026 · similarity 0.89)
• MMedAgent-RL: Optimizing Multi-Agent Collaboration for Multimodal Medical Reasoning (ICLR 2026 · similarity 0.89)
• Spotlight on Token Perception for Multimodal Reinforcement Learning (ICLR 2026 · similarity 0.88)
• PeRL: Permutation-Enhanced Reinforcement Learning for Interleaved Vision-Language Reasoning (NeurIPS 2025 · similarity 0.88)
• MedVR: Annotation-Free Medical Visual Reasoning via Agentic Reinforcement Learning (ICLR 2026 · similarity 0.88)
• QoQ-Med: Building Multimodal Clinical Foundation Models with Domain-Aware GRPO Training (NeurIPS 2025 · similarity 0.88)