Research map / Detection and segmentation / Open-set and few-shot detection

Advancing Generalizable Tumor Segmentation with Anomaly-Aware Open-Vocabulary Attention Maps and Frozen Foundation Diffusion Models

CVPR 2025 · Detection and segmentation › Open-set and few-shot detection · approach: Open-vocabulary semantic segmentation · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DiffuGTS, which uses internal features of a frozen medical foundation diffusion model (MAISI) with anomaly-aware open-vocabulary attention maps and training-free latent inpainting plus pixel- and feature-level residual learning to segment unseen tumors zero-shot.

Problem

Zero-shot tumor segmentation across diverse anatomical regions and imaging modalities, where vision-language and SAM-based methods give limited segmentation quality and generalization.

Approach

Repurpose frozen MAISI VAE encoder features with text-prompt cross-attention to form anomaly-aware open-vocabulary attention maps, then refine masks by inpainting pseudo-healthy images with the frozen diffusion model and computing pixel-level and feature-level residuals.

Key results

• Leave-one-out zero-shot tumor segmentation: DiffuGTS reaches 59.80% DSC on kidney tumor (KiTS23) and 63.23% DSC on MSD liver tumor, outperforming ZePT and Malenia by at least 4.74% DSC.
• Unseen MRI modality setting: DiffuGTS achieves 50.31% DSC on in-house MRI liver tumor, surpassing the next best method by at least 21.28% DSC.
• Replacing AOVA maps with a Mask2Former backbone drops DSC on unseen MSD brain tumor from 47.51% to 36.57%.

Measured effects reported in the paper

Surprising findings

• The frozen-diffusion pipeline has about 12,876 GFLOPs versus roughly 3,887 GFLOPs for ZePT, which the authors flag as a substantial computational limitation.
• Fine-tuning the MAISI VAE encoder performs only slightly better than using it without an adapter, and both are far below the adapter variant.

Limitations

• High FLOPs from running the diffusion model for feature extraction and inpainting.
• Relies on MAISI as the frozen foundation model; generalization beyond its 127-structure conditioning is not tested.
• Evaluation covers only 7 tumor categories across four datasets with some private data.

Experimental setup

Datasets: MSD KiTS23 BraTS23 TotalSegmentator in-house MRI liver tumor dataset

Benchmarks: MSD KiTS23 BraTS23 MSD-Brain in-house MRI liver tumor dataset

Metrics: DSC NSD

Baselines: SAM 2 SaLIP H-SAM ZePT Malenia DDPM-MAD MVFA THOR

Builds on: MAISI CLIP RePaint Mask2Former

Closest papers by meaning

• ZePT: Zero-Shot Pan-Tumor Segmentation via Query-Disentangling and Self-Prompting (CVPR 2024 · similarity 0.89)
• DiffCut: Catalyzing Zero-Shot Semantic Segmentation with Diffusion Features and Recursive Normalized Cut (NeurIPS 2024 · similarity 0.85)
• Why Invariance is Not Enough for Biomedical Domain Generalization and How to Fix It (NeurIPS 2026 · similarity 0.85)
• Beyond ViT Tokens: Masked-Diffusion Pretrained Convolutional Pathology Foundation Model for Cell-Level Dense Prediction (NeurIPS 2026 · similarity 0.85)
• Adapting Visual-Language Models for Generalizable Anomaly Detection in Medical Images (CVPR 2024 · similarity 0.85)
• LangDAug: Langevin Data Augmentation for Multi-Source Domain Generalization in Medical Image Segmentation (ICML 2025 · similarity 0.84)