Research map / Vision-language and multimodal / Human and embodied modeling
Better Tokens for Better 3D: Advancing Vision-Language Modeling in 3D Medical Imaging
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BTB3D, a causal 3D convolutional encoder-decoder with wavelet compression and lookup-free quantization, trained in three stages to tokenize long CT volumes for report generation and text-to-CT synthesis.
Problem
3D medical vision-language modeling struggles with long, high-resolution CT volumes: contrastive vision encoders are misaligned with clinical language, and slice-wise or lightweight encoder-decoders lose fine anatomy for report generation and text-to-CT synthesis.
Approach
A causal convolutional encoder-decoder with Haar wavelet input compression and lookup-free quantization, trained with reconstruction on short slice excerpts, then overlapping-window tiling and decoder-only long-sequence fine-tuning so it generalizes to 300+ slice scans.
Key results
- Ours-16 reaches F1 0.258 vs 0.184 for CT-CHAT on CT-RATE (about 40% relative gain) and BLEU-1 0.439 vs 0.373.
- Ours-8 reduces mean FID to 2.236 vs 9.512 for GenerateCT (about 76% reduction); FVD with I3D is 325.5 vs 1512.5.
- Three-stage training raises 8x8x8 reconstruction PSNR from 9.35 (Stage 1) to 23.98 (Stage 2) and 28.17 (Stage 3).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use BTB3D 16²×8 volumetric tokens instead of CT-CHAT's vision encoder | Report generation with LLaMA 3.1-8B · CT-RATE | F1: 0.184 -> 0.258 | Helped |
| use BTB3D 16²×8 volumetric tokens instead of the best baseline | Report generation with LLaMA 3.1-8B · Rad-ChestCT | F1: 0.182 -> 0.266 | Helped |
| use BTB3D 8³ instead of GenerateCT | Text-conditional CT generation · CT-RATE | Mean FID: 9.512 -> 2.236 | Helped |
| use BTB3D 8³ instead of GenerateCT | Text-conditional CT generation · CT-RATE | FVD (CT-Net): 7.659 -> 3.955 | Helped |
| increase compression from 8³ to 16²×8 (ablation) | BTB3D report generation · CT-RATE | F1: 0.187 -> 0.258 | Helped |
| increase compression from 8³ to 16²×8 (ablation) | BTB3D text-conditional CT generation · CT-RATE | Mean FID: 2.236 -> 5.011 | Hurt |
| add Stage 2 overlapping temporal tiling after Stage 1 (ablation) | BTB3D 8³ reconstruction · CT-RATE | PSNR: 9.350 -> 23.980 | Helped |
| add Stage 3 decoder-only long-sequence fine-tuning after Stage 2 (ablation) | BTB3D 8³ reconstruction · CT-RATE | PSNR: 23.980 -> 28.166 | Helped |
| add Stage 2 overlapping temporal tiling after Stage 1 (ablation) | BTB3D 16²×8 reconstruction · CT-RATE | PSNR: 11.067 -> 23.808 | Helped |
| add Stage 3 decoder-only long-sequence fine-tuning after Stage 2 (ablation) | BTB3D 16²×8 reconstruction · CT-RATE | PSNR: 23.808 -> 26.750 | Helped |
Surprising findings
- The higher-compression 16x16x8 variant gives better report generation while the 8x8x8 variant gives better text-to-CT synthesis, showing a task-dependent compression trade-off.
- Merlin was retrained on CT-RATE because its original weights are private, and it achieves high precision but low recall in report generation.
Limitations
- Evaluated only on chest CT; other anatomies and modalities are untested.
- No repeated runs or error bars, so statistical significance is not established.
- Not assessed by expert reader studies or in clinical workflows.
Experimental setup
Datasets: CT-RATE RadChestCT
Benchmarks: CT-RATE RAD-ChestCT
Metrics: F1 precision recall CRG BLEU METEOR PSNR SSIM MSE FID
Baselines: CT2Rep CT-CHAT Merlin GenerateCT MedSyn
Builds on: CT-CHAT GenerateCT lookup-free quantization CT-ViT
Closest papers by meaning
- Enhancing vision-language models for medical imaging: bridging the 3D gap with innovative slice selection (NeurIPS 2024 · similarity 0.90)
- 3DMedAgent: Unified Perception-to-Understanding for 3D Medical Analysis (ICML 2026 · similarity 0.88)
- Large-scale and Fine-grained Vision-language Pre-training for Enhanced CT Image Understanding (ICLR 2025 · similarity 0.88)
- Autoregressive Sequence Modeling for 3D Medical Image Representation (AAAI 2025 · similarity 0.88)
- Photon: Speedup Volume Understanding with Efficient Multimodal Large Language Models (ICLR 2026 · similarity 0.88)
- VISTA3D: A Unified Segmentation Foundation Model For 3D Medical Imaging (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.