Research map / 3D vision / 3D generation
Raptor: Scalable Train-Free Embeddings for 3D Medical Volumes Leveraging Pretrained 2D Foundation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Raptor, a train-free method that embeds 3D medical volumes by running a frozen DINOv2-L over axial, coronal and sagittal slices, mean-pooling across slices, and compressing patch tokens with random projections.
Problem
Learning embeddings for 3D medical volumes (e.g., MRI, CT) is computationally expensive and limited by scarce large 3D datasets, so existing volumetric foundation models require heavy pretraining.
Approach
Use a frozen 2D image foundation model on cross-sections from three orthogonal views, average slices within each view, and compress the patch-token tensor with a Gaussian random projection, yielding a training-free, sub-cubic volume embedding fed to a linear or MLP predictor.
Key results
- Raptor (K=100) reaches average classification accuracy about 3% above SuPreM and 13% above SLIViT across classification datasets, per the paper's averaged gains.
- On UK Biobank brain IDP regression, Raptor averages r^2 0.389 versus 0.299 for SuPreM and 0.313 for Merlin.
- Raptor-B (K=10) matches SuPreM's average classification accuracy with a 28.8x smaller embedding.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Raptor instead of the strongest listed baseline | 3D MedMNIST Organ · Organ | AUROC: 0.999 vs 0.999 | No clear effect |
| Use Raptor instead of the strongest listed baseline | 3D MedMNIST Nodule · Nodule | AUROC: 0.920 -> 0.929 | Helped |
| Use Raptor instead of the strongest listed baseline | 3D MedMNIST Fracture · Fracture | AUROC: 0.759 -> 0.677 | Hurt |
| Use Raptor instead of the strongest listed baseline | 3D MedMNIST Adrenal · Adrenal | AUROC: 0.913 -> 0.926 | Helped |
| Use Raptor instead of the strongest listed baseline | 3D MedMNIST Vessel · Vessel | AUROC: 0.964 -> 0.966 | Helped |
| Use Raptor instead of the strongest listed baseline | 3D MedMNIST Synapse · Synapse | AUROC: 0.918 -> 0.943 | Helped |
| Use Raptor instead of the strongest listed baseline | CC-CCII · CC-CCII | AUROC: 0.988 -> 0.997 | Helped |
| Use Raptor instead of the strongest listed baseline | CTRG-C · CTRG-C | AUROC: 0.613 -> 0.620 | Helped |
| Use Raptor instead of the strongest listed baseline | CTRG-B · CTRG-B | AUROC: 0.717 -> 0.711 | Hurt |
| Use Raptor instead of Raptor-B (K=10) (ablation) | UKBB Brain IDP regression · UKBB Brain IDP (Avg.) | r^2: 0.356 -> 0.389 | Helped |
| Increase projections from K=1 to K=100 (ablation) | Raptor on 3D MedMNIST · 3D MedMNIST | average AUROC: 0.818 -> 0.899 | Helped |
| Increase projections from K=100 to K=150 (ablation) | Raptor on 3D MedMNIST · 3D MedMNIST | average AUROC: 0.899 -> 0.900 | No clear effect |
Surprising findings
- ResNet trained from scratch outperforms several pretrained medical volume models on UK Biobank brain regression, suggesting CT-based volumetric pretraining transfers poorly to MRI.
- Raptor performs poorly on 3D FractureMNIST, which the authors attribute to negative slice-wise alpha values that cancel distances during averaging.
Limitations
- Modest performance on Fracture3D, attributed to slice-level misalignment that random projection and averaging can cancel out.
- Sensitivity to small digits in simulated size tasks; detection drops sharply below 32px.
- Relies on a generic 2D encoder whose features may not capture medical-specific cues; medical-domain variance only partly aligns with generic PCs.
Experimental setup
Datasets: 3D MedMNIST (six tasks) CC-CCII CTRG UK Biobank brain MRI
Benchmarks: 3D MedMNIST CC-CCII CTRG-C CTRG-B UK Biobank brain IDP regression
Metrics: AUROC accuracy r^2 AUPR
Baselines: ResNet MAE MISFM SuPreM SLIViT VoCo Merlin
Builds on: DINOv2 Johnson-Lindenstrauss lemma MedMNIST
Closest papers by meaning
- Learning General-purpose Biomedical Volume Representations using Randomized Synthesis (ICLR 2025 · similarity 0.87)
- Better Tokens for Better 3D: Advancing Vision-Language Modeling in 3D Medical Imaging (NeurIPS 2025 · similarity 0.85)
- Enhancing vision-language models for medical imaging: bridging the 3D gap with innovative slice selection (NeurIPS 2024 · similarity 0.84)
- CycleINR: Cycle Implicit Neural Representation for Arbitrary-Scale Volumetric Super-Resolution of Medical Data (CVPR 2024 · similarity 0.84)
- Knowledge Transfer Scaling Laws for 3D Medical Imaging (NeurIPS 2026 · similarity 0.84)
- Introducing 3D Representation for Dense Volume-to-Volume Translation via Score Fusion (ICML 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.