Research map / Vision-language and multimodal / Multimodal LLMs

3DMedAgent: Unified Perception-to-Understanding for 3D Medical Analysis

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Benchmarks and analysis of MLLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes 3DMedAgent, a tool-using agent that lets zero-shot 2D MLLMs answer 3D CT questions via organ-aware memory, coarse-to-fine lesion targeting, and a slice-level verification loop, and introduces the DeepChestVQA chest CT benchmark.

Problem

Existing 3D medical analysis methods are either task-specific or end-to-end 3D MLLMs that compress volumes into few tokens and generalize poorly, while 2D MLLMs cannot natively process volumetric CT.

Approach

A query-adaptive agent wraps a 2D MLLM with heterogeneous visual/text tools (VISTA3D organ masks, CT-CLIP lesion heatmaps, slice-level crop/overlay) and a long-term structured evidence memory, progressively moving from global organ statistics to regional lesion ROIs to individual slices without any 3D-specific fine-tuning.

Key results

• On the refined DeepTumorVQA subset, 3DMedAgent with GPT-5 reaches 0.66 total accuracy versus 0.41 for GPT-5 alone and 0.42 for Qwen3-VL-30B; with Qwen3-VL it reaches 0.61.
• On DeepChestVQA, 3DMedAgent (GPT-5) reaches 0.57 total accuracy versus 0.43 for the best general/medical MLLM baseline.
• Ablation on DeepTumorVQA shows the full system (OAMI+CFLT+T1S-Loop) improves total measurement/recognition/reasoning accuracy over the text-only OAMI baseline, with T1S-Loop adding gains across iterations.

Surprising findings

• 3D-specific MLLMs (M3D, RadFM) perform near random chance, below general MLLMs, suggesting overfitting to fine-tuning datasets.
• Qwen3-VL-30B is stronger than GPT-5 as a medical-reasoning baseline on DeepChestVQA, yet the agent performs better with GPT-5, implying evidence integration matters more than raw baseline strength.

Limitations

• Lesion masks are not used for memory initialization because lesion label taxonomies are inconsistent across datasets, so lesion information comes only from CT-CLIP heatmaps.
• Medical reasoning accuracy remains modest on DeepChestVQA (around 0.5), and the paper notes that residual cases demand medical understanding beyond current MLLMs.
• DeepChestVQA is small (1,020 VQA pairs from 892 CT scans), and some labels are derived from masks/rules rather than clinician annotation.

Experimental setup

Datasets: CT-RATE ReXGroundingCT NSCLC-Radiomics DeepTumorVQA

Benchmarks: DeepTumorVQA DeepChestVQA

Metrics: multiple-choice accuracy

Baselines: GPT-5 Qwen3-VL-30B MedGemma-27B HuatuoGPT-Vision-34B M3D RadFM

Builds on: CT-CLIP VISTA3D DeepTumorVQA

Closest papers by meaning

• Better Tokens for Better 3D: Advancing Vision-Language Modeling in 3D Medical Imaging (NeurIPS 2025 · similarity 0.88)
• 3D-RAD: A Comprehensive 3D Radiology Med-VQA Dataset with Multi-Temporal Analysis and Diverse Diagnostic Tasks (NeurIPS 2025 · similarity 0.88)
• Large-scale and Fine-grained Vision-language Pre-training for Enhanced CT Image Understanding (ICLR 2025 · similarity 0.87)
• Med-Scout: Curing MLLMs' Geometric Blindness in Medical Perception via Geometry-Aware RL Post-Training (ICML 2026 · similarity 0.87)
• How Do Medical MLLMs Fail? A Study on Visual Grounding in Medical Images (ICLR 2026 · similarity 0.86)
• AOR: Anatomical Ontology-Guided Reasoning for Medical Large Multimodal Model in Chest X-Ray Interpretation (NeurIPS 2025 · similarity 0.86)