Research map / Detection and segmentation / Open-set and few-shot detection

Benchmarking Segmentation Models with Mask-Preserved Attribute Editing

CVPR 2024 · Detection and segmentation › Open-set and few-shot detection · approach: Amodal and open-world instance segmentation · 4 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Builds a mask-preserved diffusion-based attribute editing pipeline that edits object and image attributes of Pascal VOC validation images, creating the Pascal-EA benchmark to evaluate robustness of close-set and open-vocabulary segmentation models.

Problem

Robustness evaluation of segmentation models has focused on global image attribute shifts (e.g., weather), leaving local object attributes (color, material, pattern) under-explored, partly because re-annotating segmentation masks for edited images is costly.

Approach

Mask-guided attention in a latent diffusion model (rectifying self- and cross-attention with object masks, plus ControlNet) edits text-specified attributes while preserving the original segmentation layout, so original labels can be reused for edited images without tuning.

Key results

• Material variations cause the largest drops (18.74% to 31.22% mIoU decline), versus 8.27% to 12.01% for color; sketch style drops 56.53%.
• Open-vocabulary CATSeg (95.59 val mIoU, 82.98 under variations) is more robust than ODISE (69.79) and SEEM with SAM-B backbone (67.19), showing stronger backbones do not guarantee robustness.
• Mask-guided attention plus ControlNet achieves the lowest DINO Dist (0.003 material, 0.001 color) and LPIPS (0.185 material) among compared editing methods.

Measured effects reported in the paper

Surprising findings

• Transformer-based close-set models (Segformer, Mask2Former) show larger overall performance declines than CNN-based DeepLabV3+ and OCRNet, contrary to prior findings that transformers are more robust.
• Stronger backbones and more training data (e.g., SEEM with SAM backbone, ODISE) do not translate into better robustness under attribute variations.

Limitations

• Diffusion failure modes make editing person attributes difficult.
• Inherent diffusion biases can alter unintended attributes, e.g., changing material to wood may shift color to brown.
• Evaluation limited to Pascal VOC object-centric images and a manually chosen set of attributes.

Experimental setup

Datasets: Pascal VOC Cityscapes ACDC Stylized COCO

Benchmarks: Pascal-EA

Metrics: mIoU LPIPS DINO Dist CLIP Acc FID

Baselines: DeepLabV3+ OCRNet Segmenter Segformer Mask2Former CATSeg OVSeg ODISE X-Decoder SEEM CutOut CutMix

Builds on: Plug-and-Play (PnP) Latent Diffusion Model ControlNet Prompt-to-Prompt Blended Latent Diffusion

Closest papers by meaning

• Uncertainty-aware Fine-tuning of Segmentation Foundation Models (NeurIPS 2024 · similarity 0.87)
• From Pixels to Concepts: Do Segmentation Models Understand What They Segment? (NeurIPS 2026 · similarity 0.85)
• Early Semantic Grounding in Image Editing Models for Zero-Shot Referring Image Segmentation (NeurIPS 2026 · similarity 0.85)
• RobustSAM: Segment Anything Robustly on Degraded Images (CVPR 2024 · similarity 0.85)
• Test-Time Adaptation of Vision-Language Models for Open-Vocabulary Semantic Segmentation (NeurIPS 2025 · similarity 0.85)
• ValUES: A Framework for Systematic Validation of Uncertainty Estimation in Semantic Segmentation (ICLR 2024 · similarity 0.84)