Research map / Image generation / Text-to-image diffusion
FreeGen: Bridging Visual-Linguistic Discrepancies Towards Diffusion-based Pixel-level Data Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FreeGen, a real-data-free pipeline that uses Stable Diffusion attention rectification, a temporal fluctuation filter, and a self-trained grounding decoder to synthesize open-vocabulary pixel-level segmentation data.
Problem
Cross-attention masks from text-to-image diffusion models are coarse and unstable (visual-linguistic discrepancies), giving poor pseudo-masks for automatic segmentation data synthesis, especially open-vocabulary.
Approach
Learns an Attention Self-Rectification mechanism that reweights cross-attention with a learnable cross-head interaction of self-attention, filters synthetic masks with a Temporal Fluctuation Factor over sampling timesteps, and trains an Entity-level Grounding Decoder with hardness-aware self-training for open-vocabulary masks.
Key results
- Mask2Former Swin-B trained on FreeGen synthetic data reaches 76.3 mIoU on VOC 2012 val, versus 70.6 for DiffuMask and 73.7 for DatasetDM.
- On zero-shot VOC 2012 with text supervision, FreeGen (Swin-B) reaches 64.9 unseen mIoU, the best reported unseen score versus 63.6 for ZegFormer.
- Ablation on VOC 2012 shows the raw attention baseline at 60.5 mIoU, rising to 67.1 with ASR and 69.6 with the full pipeline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FreeGen synthetic data instead of Dataset Diffusion | DeepLabV3 R50 · VOC 2012 val | mIoU: 64.8 -> 67.5 | Helped |
| Use FreeGen synthetic data instead of DatasetDM | Mask2former Swin-B · VOC 2012 val | mIoU: 73.7 -> 76.3 | Helped |
| Use FreeGen synthetic data instead of Dataset Diffusion | DeepLabV3 R101 · COCO 2017 val | mIoU: +3.2%∼+3.8% | Helped |
| Use FreeGen synthetic data instead of DiffuMask | Mask2former Swin-B · Cityscapes val, Vehicle class | mIoU: 0.7% | Helped |
| Use FreeGen for open-vocabulary training | VOC 2012 zero-shot segmentation · VOC 2012 val, unseen classes | mIoU: 64.9 v.s 63.6 | Helped |
| Use FreeGen rather than DatasetDM for domain-generalization training | Mask2former R50 · VOC 2012 val, domain generalization | mIoU: 80.0; 6.4% | Helped |
| Add SAM for primary mask generation (ablation) | FreeGen with Mask2former R50 · VOC 2012 val | mIoU: 66.2 -> 69.6 | Helped |
| Add hardness-aware self-training (ablation) | FreeGen with Mask2former R50 · VOC 2012 val | mIoU: 68.4 -> 69.6 | Helped |
| Add the Entity-level Grounding Decoder and hardness-aware self-training (ablation) | FreeGen with Mask2former R50 · VOC 2012 val | mIoU: 67.1 -> 69.6 | Helped |
| Add the Temporal Fluctuation Factor, Entity-level Grounding Decoder, and hardness-aware self-training (ablation) | FreeGen with Mask2former R50 · VOC 2012 val | mIoU: 66.4 -> 69.6 | Helped |
| Add Attention Self-Rectification, Temporal Fluctuation Factor, Entity-level Grounding Decoder, and hardness-aware self-training (ablation) | FreeGen with Mask2former R50 · VOC 2012 val | mIoU: 63.8 -> 69.6 | Helped |
| Add Attention Self-Rectification, Temporal Fluctuation Factor, Entity-level Grounding Decoder, hardness-aware self-training, and SAM (ablation) | FreeGen with Mask2former R50 · VOC 2012 val | mIoU: 60.5 -> 69.6 | Helped |
Surprising findings
- Without SAM the method still reaches 66.2 mIoU on VOC 2012, indicating the generative strategy contributes beyond SAM masks.
Limitations
- Performance remains below real-data training (69.6 vs 77.2 mIoU for Mask2Former R50 on VOC 2012).
- Relies on SAM for initial pseudo-masks and on multiple synthetic generation steps, adding pipeline complexity.
Experimental setup
Datasets: VOC 2012 SBD COCO 2017 Cityscapes LAION5B
Benchmarks: VOC 2012 COCO 2017 Cityscapes ADE20K
Metrics: mIoU harmonic mIoU seen mIoU unseen mIoU
Baselines: DiffuMask DatasetDM Dataset Diffusion Attn2Mask ZS3 CaGNet SIGN ZegFormer STRICT Li et al.
Builds on: Stable Diffusion DiffuMask Dataset Diffusion Mask2Former SAM
Closest papers by meaning
- Open-Vocabulary Attention Maps with Token Optimization for Semantic Segmentation in Diffusion Models (CVPR 2024 · similarity 0.88)
- Seg4Diff: Unveiling Open-Vocabulary Semantic Segmentation in Text-to-Image Diffusion Transformers (NeurIPS 2025 · similarity 0.88)
- Training-Free Open-Vocabulary Segmentation with Offline Diffusion-Augmented Prototype Generation (CVPR 2024 · similarity 0.88)
- Localizing and Editing Knowledge In Text-to-Image Generative Models (ICLR 2024 · similarity 0.88)
- Leveraging Unpaired Data for Vision-Language Generative Models via Cycle Consistency (ICLR 2024 · similarity 0.88)
- Progressive Text-to-Image Diffusion with Soft Latent Direction (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.