Research map / Vision-language and multimodal / Multimodal LLMs

3D-RAD: A Comprehensive 3D Radiology Med-VQA Dataset with Multi-Temporal Analysis and Diverse Diagnostic Tasks

NeurIPS 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces 3D-RAD, a large-scale 3D chest CT Med-VQA benchmark with six tasks including multi-temporal diagnosis, plus a 136K-sample training set, and shows fine-tuning M3D models on it substantially boosts performance.

Problem

Existing Med-VQA datasets are mostly 2D, have limited task diversity, and lack temporal/longitudinal reasoning, hindering evaluation and training of 3D medical VLMs.

Approach

Builds 3D Med-VQA QA pairs from CT-RATE chest CT scans and reports using templates and GPT-4o-mini generation, with temporal QA derived from multi-phase binary labels, then filters with LLM scoring and human validation; fine-tunes 3D Med-VLMs on the resulting training set.

Key results

• Fine-tuning M3D (Phi3-4B) on 3D-RAD raised Task 6 longitudinal temporal diagnosis accuracy from 24.31% to 74.77% and Task 4 existence detection from 40.25% to 82.43%.
• Zero-shot models performed poorly on temporal tasks, with the best (RadFM) reaching 44.11% on static temporal diagnosis and 42.99% on longitudinal diagnosis.
• Human validation found 96.17% (577/600) factual agreement with radiology reports after filtering.

Measured effects reported in the paper

Surprising findings

• Zero-shot GPT-4.1 scored 68.05% on Task 6 while the fine-tuned-ready medical VLMs were near 24%, showing general models can exploit label sequences better than medical VLMs.
• Medical computation (Task 3) retained low BLEU/ROUGE even after fine-tuning despite high BERTScore, indicating persistent numerical reasoning limits.

Limitations

• Longitudinal task uses only binary label histories, not full multi-scan 3D volumes, as current architectures cannot take multiple volumes.
• Limited to chest CT from the CT-RATE source with 18 label categories.
• No open-ended questions for the longitudinal temporal task.

Experimental setup

Datasets: CT-RATE 3D-RAD-T 3D-RAD-Bench

Benchmarks: 3D-RAD-Bench

Metrics: accuracy BLEU ROUGE BERTScore

Baselines: RadFM M3D (Llama2-7B) M3D (Phi3-4B) OmniV LLaVA-OneVision Qwen2.5-VL Gemma-3 InternVL-3 Gemini-2.0-flash GPT-4.1

Builds on: M3D CT-RATE RadFM

Closest papers by meaning

• Falcon Medical Visual Question Answering (AAAI 2025 · similarity 0.90)
• OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLM (CVPR 2024 · similarity 0.89)
• MedAtlas: Evaluating LLMs for Multi-Round, Multi-Task Medical Reasoning Across Diverse Imaging Modalities and Clinical Text (AAAI 2026 · similarity 0.89)
• MediConfusion: Can you trust your AI radiologist? Probing the reliability of multimodal medical foundation models (ICLR 2025 · similarity 0.88)
• 3DMedAgent: Unified Perception-to-Understanding for 3D Medical Analysis (ICML 2026 · similarity 0.88)
• Alignment, Mining and Fusion: Representation Alignment with Hard Negative Mining and Selective Knowledge Fusion for Medical Visual Question Answering (CVPR 2025 · similarity 0.88)