Research map / Vision-language and multimodal / Multimodal LLMs

Photon: Speedup Volume Understanding with Efficient Multimodal Large Language Models

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Training-free token reduction for MLLMs

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Photon is a 3D-native medical MLLM that prunes visual tokens per instance using instruction-conditioned saliency and a predicted threshold, with a surrogate-gradient scheme so pruning stays trainable.

Problem

Scaling multimodal LLMs to full 3D medical volumes is computationally expensive; slice sampling and fixed-length token compression lose volumetric detail, and existing adaptive pruning either applies only at inference or is not trainable through discrete token dropping.

Approach

Instruction-conditioned Token Scheduling predicts an instance-specific retention threshold from instruction-vision saliency and hard-drops tokens in the forward pass, while Surrogate Gradient Propagation routes straight-through gradients to retention probabilities so the threshold predictor is trained during fine-tuning.

Key results

• On 3D-RAD finetuned, Photon (3B) reaches 83.07 existence detection accuracy vs 81.09 for the best finetuned baseline (M3D-L2 7B), and 52.86 static temporal diagnosis vs 51.20.
• On DeepTumorVQA, Photon (3B) achieves total average 0.686 multi-choice and 0.619 free-text, versus 0.662 and 0.555 for the best prior baselines.
• Photon keeps about 0.39K visual tokens on average on 3D-RAD (vs 7.0K for Qwen2.5-VL), with inference speed 4.12 tokens/s vs 2.30 for finetuned Qwen2.5-VL.

Surprising findings

• Fully finetuning the vision encoder and MLP after alignment caused overfitting and loss of instruction-following, collapsing accuracy on several tasks.
• HiPrune slows decoding despite keeping fewer tokens, and VisionZip/HiPrune gave little inference speedup in this setting.

Limitations

• Gains in inference speed are modest due to KV cache usage.
• Errors concentrate on fine-grained quantitative estimation of small lesions and rare anatomical patterns.
• Pruning baselines were compared only at inference time on a Qwen2.5-VL model trained without pruning, which may not reflect their full training recipes.

Experimental setup

Datasets: CT-RATE AbdomenAtlas 3.0 3D-RAD DeepTumorVQA MedFrameQA SLAKE NExT-QA

Benchmarks: 3D-RAD DeepTumorVQA MedFrameQA SLAKE NExT-QA

Metrics: accuracy BLEU ROUGE BERT Score MRA tokens/s peak GPU memory

Baselines: Qwen2.5-VL RadFM M3D-L2 M3D-P3 OmniV-Med Lingshu Merlin CT-CHAT VisionZip HiPrune

Builds on: Qwen2.5-VL VisionZip ATP-LLaVA FlashAttention2

Closest papers by meaning

• Towards a Multimodal Large Language Model with Pixel-Level Insight for Biomedicine (AAAI 2025 · similarity 0.89)
• Better Tokens for Better 3D: Advancing Vision-Language Modeling in 3D Medical Imaging (NeurIPS 2025 · similarity 0.88)
• MIMO: A Medical Vision Language Model with Visual Referring Multimodal Input and Pixel Grounding Multimodal Output (CVPR 2025 · similarity 0.87)
• Zero-shot 3D Question Answering via Voxel-based Dynamic Token Compression (CVPR 2025 · similarity 0.87)
• Falcon Medical Visual Question Answering (AAAI 2025 · similarity 0.87)
• 3D-RAD: A Comprehensive 3D Radiology Med-VQA Dataset with Multi-Temporal Analysis and Diverse Diagnostic Tasks (NeurIPS 2025 · similarity 0.87)