Research map / Vision-language and multimodal / CLIP and zero-shot
VOILA: Complexity-Aware Universal Segmentation of CT Images by Voxel Interacting with Language
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VOILA, a universal CT segmentation method that classifies voxels by cosine similarity to CLIP text embeddings, with a voxel-wise F1 loss and a CVAE-based complexity-aware sampler that focuses training on hard voxels.
Problem
Universal multi-class CT segmentation with text prompts suffers from information-density imbalance between 3D images and text, the poor scaling of fully connected classification layers, class imbalance between foreground and background, and weak cross-dataset generalization.
Approach
Replace the fully connected classifier with voxel-to-text cosine similarity against frozen CLIP text embeddings (with neighbor-structure prompts), use a voxel-wise F1 loss for imbalance, and sample voxels via a CVAE that generates pseudo complexity heatmaps from a Gaussian mixture.
Key results
- VOILA reaches 92.1 average Dice on TotalSegmentator v2 with 6.44M trainable parameters, versus 87.9 for nnUNet (22.68M) and 87.0 for UNETR++ (42.97M).
- Trained on Ts-v2 and evaluated without fine-tuning, VOILA gets 81.4 Dice / 80.9 NSD / 16.0 HD95 on BTCV, improving NSD by 9 points over its supervised counterpart's baseline shown in the table.
- Sampling 10% of voxels with CAS gives 92.1 Dice on Ts-v2, versus 86.0 without sampling and 91.5 with random sampling at the same ratio.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add VOILA instead of nnUNet | Universal CT segmentation · Ts-v2 | Dice: 87.9 -> 92.1 | Helped |
| Add VOILA instead of nnUNet | Universal CT segmentation · WORD | Dice: 81.1 -> 83.0 | Helped |
| Add VOILA instead of UNETR++ | Universal CT segmentation · BTCV | Dice: 73.4 -> 74.1 | Helped |
| Add VOILA instead of UNETR++ | Universal CT segmentation · AMOS | Dice: 82.5 -> 83.4 | Hurt |
| Use VOILA instead of UNETR++ | Universal CT segmentation · LiTS | Dice: 94.8 -> 91.9 | Hurt |
| Use CAS sampling at ratio 0.1 rather than no sampling (ablation) | VOILA · Ts-v2 | Dice: 86.0 -> 92.1 | Helped |
| Use CAS sampling at ratio 0.1 rather than ratio 0.01 (ablation) | VOILA · Ts-v2 | Dice: 88.6 -> 92.1 | Helped |
| Use CAS sampling at ratio 0.1 rather than ratio 0.3, 0.5, or 0.7 (ablation) | VOILA · Ts-v2 | Dice: 91.5 -> 92.1; 90.6 -> 92.1; 88.1 -> 92.1 | Helped |
| Replace CAS with random sampling at ratio 0.1 (ablation) | VOILA · Ts-v2 | Dice: 92.1 -> 91.5 | Hurt |
| Replace CAS with random sampling at ratio 0.1 (ablation) | VOILA · BTCV | Dice: 81.4 -> 79.9 | Hurt |
| Replace CAS with random sampling at ratio 0.1 (ablation) | VOILA · WORD | Dice: 81.2 -> 80.4 | Hurt |
| Remove sampling (ablation) | VOILA · BTCV and WORD | Dice: BTCV: 73.0 -> 81.4; WORD: 73.5 -> 81.2 | Hurt |
Surprising findings
- VOILA underperforms on single-organ datasets such as Pancreas-CT (73.3 Dice) and LiTS (91.9 Dice) relative to some baselines.
- Without fine-tuning, VOILA's Dice on WORD (81.2) and AMOS (77.1) is lower than its supervised results, even though NSD and HD95 improve on other datasets.
Limitations
- Performance lags on single-organ datasets such as Pancreas-CT and LiTS.
- Generalization without fine-tuning is mixed: Dice drops on WORD and AMOS, with gains mainly in NSD and HD95.
- Only three baselines compared, and results are restricted to a fixed 400-epoch budget.
Experimental setup
Datasets: TotalSegmentator v2 BTCV Pancreas-CT WORD LiTS AbdomenCT-1K AMOS
Benchmarks: TotalSegmentator v2 BTCV Pancreas-CT WORD LiTS AbdomenCT-1K AMOS
Metrics: Dice NSD HD95
Baselines: nnUNet UNETR++ nnFormer
Builds on: CLIP InfoNCE UNETR++ Dice loss
Closest papers by meaning
- COS3D: Collaborative Open-Vocabulary 3D Segmentation (NeurIPS 2025 · similarity 0.86)
- Universal Segmentation at Arbitrary Granularity with Language Instruction (CVPR 2024 · similarity 0.85)
- SegVol: Universal and Interactive Volumetric Medical Image Segmentation (NeurIPS 2024 · similarity 0.85)
- Better Tokens for Better 3D: Advancing Vision-Language Modeling in 3D Medical Imaging (NeurIPS 2025 · similarity 0.84)
- USE: Universal Segment Embeddings for Open-Vocabulary Image Segmentation (CVPR 2024 · similarity 0.84)
- Segment Any 3D Object with Language (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.