Research map / Detection and segmentation / Weakly supervised detection and segmentation

Iterative Self-Training with Class-Aware Text-to-Image Synthesis for Visual Task Learning

AAAI 2025 · Detection and segmentation › Weakly supervised detection and segmentation · approach: Semi-supervised instance segmentation · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes IST-CATS, which uses LLM-driven class-aware prompt chains to drive latent diffusion image synthesis with pseudo-labels, then iteratively self-trains segmentation and detection models with a class-adaptive label filter (LabFilt).

Problem

Synthetic images from text-to-image diffusion models have limited diversity, noisy pseudo labels, and domain gaps to real images, which hurt downstream object detection and semantic segmentation.

Approach

A class-aware chain of LLM prompts (subcategories then detailed descriptions with clean-background cues) feeds a latent diffusion model; CutLER pseudo-labels are purified by VGG16 feature-similarity filtering, and iterative self-training on real unlabeled images uses class-adaptive pixel- and object-level thresholds (LabFilt) to refine labels across rounds.

Key results

• Semantic segmentation on PASCAL VOC 2012 reaches 59.3% mIoU, versus 58.3% for Attn2mask+BECO and 57.4% for DiffuMask.
• Object detection reaches 68.6% mAP50 on VOC 2007, 63.0% on VOC 2012, and 30.5% mAP50 on MSCOCO, surpassing prior weakly-supervised methods by 11.2, 9.1, and 2.9 points.
• Chain Generation prompts raise segmentation mIoU from 39.4% (Direct Generation) to 48.2% with DeepLabv3+ on VOC synthetic data (without CRF).

Measured effects reported in the paper

Surprising findings

• Naive mixing of simple and synthetic data improves over either alone (48.7 vs 35.4 and 39.5 mIoU), suggesting complementary diversity.

Limitations

• Iteration count is determined by monitoring training-set growth rather than a principled criterion.
• Detection model has room for improvement while segmentation plateaus after several rounds, per the authors.
• Pipeline depends on multiple components (LLM, latent diffusion, CutLER, VGG16) whose specific choices are not fully detailed.

Experimental setup

Datasets: PASCAL VOC 2012 MSCOCO PASCAL VOC 2007 Syn-VOC Syn-COCO ImageNet

Benchmarks: PASCAL VOC 2012 PASCAL VOC 2007 MSCOCO

Metrics: mIoU mAP50 AP

Baselines: DiffuMask Attn2mask DatasetDM GranSAM ST++ UniMatch CorrMatch PCL MIST CASD CBL NDI-MIL

Builds on: DiffuMask CutLER latent diffusion models ST++ DeepLabv3+

Closest papers by meaning

• Few-Shot Semantic Image Synthesis With Class Affinity Transfer (CVPR 2023 · similarity 0.87)
• InstructCV: Instruction-Tuned Text-to-Image Diffusion Models as Vision Generalists (ICLR 2024 · similarity 0.87)
• ImagineFSL: Self-Supervised Pretraining Matters on Imagined Base Set for VLM-based Few-shot Learning (CVPR 2025 · similarity 0.86)
• CtrlSynth: Controllable Image Text Synthesis for Data-Efficient Multimodal Learning (ICML 2025 · similarity 0.85)
• Text-Printed Image: Bridging the Image-Text Modality Gap for Text-centric Training of Large Vision-Language Models (CVPR 2026 · similarity 0.85)
• Beyond Words: Augmenting Discriminative Richness via Diffusions in Unsupervised Prompt Learning (CVPR 2025 · similarity 0.85)