Research map / Image generation / Text-to-image diffusion
Seg2Any: Open-set Segmentation-Mask-to-Image Generation with Precise Shape and Semantic Control
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Seg2Any, a LoRA-tuned FLUX.1-dev segmentation-mask-to-image model that decouples mask conditions into semantic attention masks and sparse entity contour maps, plus a 1M-image open-set dataset (SACap-1M) and SACap-Eval benchmark.
Problem
Segmentation-mask-to-image (S2I) generation struggles to jointly keep entity semantics aligned with regional prompts, preserve precise mask shapes, and prevent attribute leakage across entities, especially in open-set settings.
Approach
Decouple mask conditions into a Semantic Alignment Attention Mask (regional text-image binding) and a sparse Entity Contour Map fed as an extra condition modality via multimodal attention with LoRA on all branches, plus an Attribute Isolation Attention Mask that confines each entity's image tokens to themselves in middle layers.
Key results
- On SACap-Eval, Seg2Any reaches 94.90 class-agnostic MIoU versus 96.03 for real images and 84.30 for the next best (PLACE).
- On COCO-Stuff, Seg2Any achieves 45.54 MIoU, the highest among compared methods (FreestyleNet 42.42).
- On ADE20K, Seg2Any reaches 54.46 MIoU, behind PLACE (60.20) but above other baselines; its FID (32.89) is higher than PLACE (24.51).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Seg2Any instead of DreamRender | S2I generation · SACap-Eval | class-agnostic MIoU: 82.32 -> 94.90 | Helped |
| use Seg2Any instead of DreamRender | S2I generation · SACap-Eval | FID: 13.71 -> 15.53 | Hurt |
| use Seg2Any instead of PLACE | S2I generation · COCO-Stuff | MIoU: 42.23 -> 45.54 | Helped |
| use Seg2Any instead of PLACE | S2I generation · COCO-Stuff | FID: 14.95 -> 19.90 | Hurt |
| use Seg2Any instead of PLACE | S2I generation · ADE20K | MIoU: 60.20 -> 54.46 | Hurt |
| add SSFA to SAA (ablation) | Seg2Any · COCO-Stuff | MIoU: 43.48 -> 45.50 | Helped |
| add SSFA to SAA (ablation) | Seg2Any · ADE20K | MIoU: 44.85 -> 54.11 | Helped |
| add SSFA to SAA (ablation) | Seg2Any · SACap-Eval | class-agnostic MIoU: 89.39 -> 94.16 | Helped |
| add training-free AIA (ablation) | Seg2Any · SACap-Eval | color accuracy: 89.16 -> 92.16 | Helped |
| add training-free AIA (ablation) | Seg2Any · SACap-Eval | spatial accuracy: 93.43 -> 92.37 | Hurt |
| use training-based AIA instead of training-free AIA (ablation) | Seg2Any · SACap-Eval | spatial accuracy: 92.37 -> 93.89 | Helped |
| add CITF to SAA+SSFA (ablation) | Seg2Any · COCO-Stuff | MIoU: 45.50 -> 45.54 | No clear effect |
Surprising findings
- Training-free application of the attribute isolation mask introduces visual artifacts (unnatural shadows and reflections), so training is needed for it to work well.
- Seg2Any has higher FID than PLACE and FreestyleNet on closed-set benchmarks, attributed by the authors to a domain gap between FLUX and ADE20K/COCO-Stuff images.
Limitations
- Resource constraints when generating images with many entities that each have detailed descriptions.
- Regional captions from vision-language models introduce annotation noise affecting S2I training.
- Higher FID on closed-set benchmarks due to domain gap between FLUX and ADE20K/COCO-Stuff.
Experimental setup
Datasets: SACap-1M SA-1B ADE20K COCO-Stuff
Benchmarks: SACap-Eval COCO-Stuff ADE20K
Metrics: class-agnostic MIoU spatial accuracy color accuracy shape accuracy texture accuracy IR score Pick score CLIP score FID MIoU
Baselines: FreestyleNet PLACE ControlNet ControlNet++ UniControl DreamRender 3DIS EliGen GLIGEN
Builds on: FLUX OminiControl DreamRender PLACE
Closest papers by meaning
- Seg4Diff: Unveiling Open-Vocabulary Semantic Segmentation in Text-to-Image Diffusion Transformers (NeurIPS 2025 · similarity 0.89)
- Exploring Simple Open-Vocabulary Semantic Segmentation (CVPR 2025 · similarity 0.87)
- SSMG: Spatial-Semantic Map Guided Diffusion Model for Free-Form Layout-to-Image Generation (AAAI 2024 · similarity 0.86)
- Continuous, Subject-Specific Attribute Control in T2I Models by Identifying Semantic Directions (CVPR 2025 · similarity 0.86)
- Compositional Text-to-Image Synthesis with Attention Map Control of Diffusion Models (AAAI 2024 · similarity 0.86)
- SGDiff: Scene Graph Guided Diffusion Model for Image Collaborative SegCaptioning (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.