Research map / Vision-language and multimodal / Human and embodied modeling
Autoregressive Sequence Modeling for 3D Medical Image Representation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AR-SSL4M, an autoregressive pre-training framework that serializes 3D CT/MRI volumes into spatial, contrast, and semantic patch sequences and predicts next visual tokens with a random-startup prefix attention scheme.
Problem
Self-supervised representation learning for 3D medical images treats each volume as a whole and ignores relationships among local regions across images and modalities.
Approach
Transform 3D medical images into ordered patch/token sequences (spatial, contrast, semantic correlations) and pre-train a ViT with autoregressive next-token regression under prefix attention with a random start index, extending 2D prefix causal attention to 3D medical data.
Key results
- Average Dice on six MSD CT segmentation tasks reaches 0.7088, versus 0.6879 for the best medical SSL baseline (MAE3D).
- On RICORD COVID-19 classification, AUC reaches 0.9259 versus 0.8803 for MedCoSS; on LIDC-IDRI nodule classification AUC reaches 0.9361.
- LA MRI segmentation Dice is 0.9157, above the best baseline of 0.9062 (MedCoSS); with 10% labels, LA Dice is 0.8365.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the full autoregressive patch-sequence pre-training method | 3D medical image representation learning · MSD six-task average | Dice: 0.6879 -> 0.7088 | Helped |
| Use the full autoregressive patch-sequence pre-training method | 3D medical image representation learning · RICORD | AUC: 0.8803 -> 0.9259 | Helped |
| Use the full autoregressive patch-sequence pre-training method | 3D medical image representation learning · LIDC-IDRI | AUC: 0.8983 -> 0.9361 | Helped |
| Add patch-sequence transformation instead of using individual images (ablation) | the proposed method · MSD Task06 Lung | Dice: 0.5880 -> 0.6529 | Helped |
| Add random startup to autoregressive attention (ablation) | the proposed method · MSD Task06 Lung | Dice: 0.5852 -> 0.6529 | Helped |
| Add spatial sequences to pre-training (ablation) | the proposed method · MSD Task06 Lung | Dice: 0.6165 -> 0.6529 | Helped |
| Add contrast sequences to pre-training (ablation) | the proposed method · MSD Task06 Lung | Dice: 0.6076 -> 0.6529 | Helped |
| Add semantic sequences to pre-training (ablation) | the proposed method · MSD Task06 Lung | Dice: 0.6137 -> 0.6529 | Helped |
| Use stride 8 rather than stride 4 (ablation) | spatial patch-sequence construction · MSD Task06 Lung | Dice: 0.6258 -> 0.6529 | Helped |
| Use stride 8 rather than stride 4 (ablation) | spatial patch-sequence construction · RICORD | AUC: 0.8931 -> 0.9259 | Helped |
| Use stride 8 rather than stride 12 (ablation) | spatial patch-sequence construction · MSD Task06 Lung | Dice: 0.6435 -> 0.6529 | Helped |
| Use stride 8 rather than stride 12 (ablation) | spatial patch-sequence construction · RICORD | AUC: 0.8988 -> 0.9259 | Helped |
Surprising findings
- Ablation shows removing the random startup mechanism (standard autoregressive attention) lowers MSD Task06 Dice from 0.6529 to 0.5852, a larger drop than several other ablations.
Limitations
- Relies on pre-training with many public datasets; the semantic sequences depend on DeepLesion categories.
- Ablation and comparisons are reported on a limited set of tasks (only MSD Task06 and RICORD for ablations).
- Gains over medical SSL baselines on segmentation are modest (about 2.1% average Dice).
Experimental setup
Datasets: RibFrac TCIA Covid19 AMOS22 ISLES2022 AbdomenCT-1K TotalSegmentator VerSe 2020 RSNA-2022-CSFD RSNA-2020-PED STOIC FLARE22 FLARE23
Benchmarks: MSD (Task03, Task06, Task07, Task08, Task09, Task10) LA RICORD LIDC-IDRI
Metrics: Dice AUC ACC
Baselines: UNETR 3D UNet ResNet ViT-B SimCLR MoCov3 DINO PCRLv2 MAE3D MedCoSS
Builds on: masked autoencoder (MAE) prefix causal attention UNETR ViT
Closest papers by meaning
- Time-to-Event Pretraining for 3D Medical Imaging (ICLR 2025 · similarity 0.89)
- Better Tokens for Better 3D: Advancing Vision-Language Modeling in 3D Medical Imaging (NeurIPS 2025 · similarity 0.88)
- Integrating Sequence and Image Modeling in Irregular Medical Time Series Through Self-Supervised Learning (AAAI 2025 · similarity 0.88)
- Unlocking the Power of Spatial and Temporal Information in Medical Multimodal Pre-training (ICML 2024 · similarity 0.87)
- Revisiting MAE Pre-training for 3D Medical Image Segmentation (CVPR 2025 · similarity 0.87)
- Knowledge Transfer Scaling Laws for 3D Medical Imaging (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.