Research map / Label-efficient and robust learning / Test-time adaptation

Touchstone Benchmark: Are We on the Right Way for Evaluating AI Algorithms for Medical Segmentation?

NeurIPS 2024 · Label-efficient and robust learning › Test-time adaptation · approach: Characterizing out-of-distribution failures · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces Touchstone, a large-scale out-of-distribution benchmark for abdominal CT multi-organ segmentation where 19 algorithms are trained by their inventors on AbdomenAtlas 1.0 and independently evaluated by a third party on a private JHH test set (N=5,160) and public TotalSegmentator subset (N=743).

Problem

Standard medical segmentation benchmarks suffer from in-distribution and small test sets, oversimplified average metrics, unfair comparisons between proposed and baseline methods, and short-term deadlines, so benchmark results do not reliably predict real-world performance.

Approach

A third-party challenge-style benchmark where algorithm inventors train their own models on a large multi-hospital fully annotated CT dataset (AbdomenAtlas 1.0), evaluated on large held-out OOD test sets with per-class, metadata-stratified, and statistically tested analyses, plus a long-term commitment to recurring editions.

Key results

• On the private JHH test set (N=5,160), MedNeXt (89.2 DSC) and MedFormer (89.0 DSC) had the highest average DSC among nnU-Net-based and other models, with no significant difference among the top winners.
• On TotalSegmentator (N=743), STU-Net-L had the best average DSC (83.0) with STU-Net-H and ResEncL close behind.
• Per-class DSC varied by 10-20% across test sets for many algorithms, with up to 80% differences observed for SAM-Adapter on kidney.

Measured effects reported in the paper

Surprising findings

• Top-performing models were mostly CNNs in the nnU-Net framework rather than transformers, and architectural causes could not be separated from framework configuration.
• Label noise of 32.4% in the aorta annotations in AbdomenAtlas 1.0 was partially detected, while 2.6% kidney noise had small effects on performance.

Limitations

• JHH test set is private, which slows feedback and limits external reproduction.
• Predictions-only benchmark cannot isolate architecture effects from preprocessing, augmentation and hyperparameter differences.
• Training labels contain noise, especially for aorta, and TotalSegmentator test subset is not from fully unseen hospitals.

Experimental setup

Datasets: AbdomenAtlas 1.0 TotalSegmentator JHH

Benchmarks: JHH (N=5,160) TotalSegmentator (N=743)

Metrics: Dice Similarity Coefficient Normalized Surface Distance inference speed Wilcoxon signed rank test with Holm adjustment

Baselines: nnU-Net ResEncL MedNeXt STU-Net Swin UNETR UNETR SegVol MedFormer Diff-UNet

Builds on: nnU-Net AbdomenAtlas 1.0 FLARE challenge

Closest papers by meaning

• CLIMB: Data Foundations for Large Scale Multimodal Clinical Foundation Models (ICML 2025 · similarity 0.85)
• Interactive Medical Image Segmentation: A Benchmark Dataset and Baseline (CVPR 2025 · similarity 0.84)
• GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AI (NeurIPS 2024 · similarity 0.84)
• PathView-Bench: Can Multimodal Large Language Models Achieve Fine-grained Multiscale Understanding of Pathology Images? (NeurIPS 2026 · similarity 0.84)
• U2-BENCH: Benchmarking Large Vision-Language Models on Ultrasound Understanding (ICLR 2026 · similarity 0.84)
• Integrating Deep Metric Learning with Coreset for Active Learning in 3D Segmentation (NeurIPS 2024 · similarity 0.84)