Research map / Image generation / Autoregressive and latent generation
WinSyn: : A High Resolution Testbed for Synthetic Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces WinSyn, a high-resolution real window photo dataset with a procedural window generator, and benchmarks synthetic-to-real segmentation transfer using a BEiTv2 model.
Problem
Closing the synthetic-to-real domain gap for procedurally generated training data, using window facade semantic segmentation as a proxy task, where existing architectural datasets are low-resolution and small.
Approach
A high-resolution (4K-6K) real window dataset paired with a split-grammar procedural window generator (CGA shape grammar plus Bezier-curve window shapes, rendered in Blender Cycles), used to measure how far synthetic-trained segmentation falls short of real-trained performance and which generator factors matter.
Key results
- Baseline synthetic-trained BEiTv2 reaches 32.58 mIoU on real test images versus 58.69 for real-trained, showing a large domain gap.
- Synthetic data saturates with more samples while real data keeps improving; no synthetic volume closed the gap.
- Material restriction caused at least an 18% mIoU drop; camera radius and window geometry had limited impact (about 1-6%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train on synthetic images instead of global real images | BEiT base segmentation · Global real test split | mIoU: 53.79 -> 31.23 | Hurt |
| Increase rendering samples per pixel (ablation) | Baseline procedural model · Segmentation on real test images | mIoU: 68% | Helped |
| Restrict or simplify the materials (ablation) | Baseline procedural model · Segmentation on real test images | mIoU: 18% | Hurt |
| Vary the lighting model (ablation) | Baseline procedural model · Segmentation on real test images | mIoU: 15.35% | Mixed |
| Use daylight-only lighting (ablation) | Baseline procedural model · Segmentation on real test images | mIoU: 1.04% | Hurt |
| Vary window dimensions and shapes (ablation) | Baseline procedural model · Segmentation on real test images | mIoU: 5.7% | Mixed |
| Use the best-performing tested window geometry (ablation) | Baseline procedural model · Segmentation on real test images | mIoU: 1.2% | Hurt |
| Vary camera position (ablation) | Baseline procedural model · Segmentation on real test images | mIoU | No clear effect |
| Add smaller semantic classes to the procedural model (ablation) | Baseline procedural model · Segmentation on real test images | mIoU | Mixed |
| Add 2,048 synthetic samples to a small real dataset | BEiT base segmentation · Real-world training data, including n = 152 | mIoU: 44.96 | Helped |
| Add synthetic samples when training with larger real datasets | BEiT base segmentation · Real-world training data | mIoU | Hurt |
Surprising findings
- Adding more synthetic data does not help beyond a point; no volume of synthetic data overcomes the domain gap.
- Label adaptation, a method using real labels, is needed for synthetic data to beat real data in prior work, which this paper does not rely on.
Limitations
- Substantial synthetic-to-real mIoU gap remains (about 32 vs 59 mIoU), so the procedural model does not yet match real data.
- Synthetic data saturates and adding it does not help large real datasets.
- Evaluation limited to window-focused semantic segmentation with one main architecture.
Experimental setup
Datasets: WinSyn real window photos WinSyn synthetic windows from procedural model
Benchmarks: WinSyn real test set (4,906 real images)
Metrics: mIoU Spearman rank correlation
Baselines: DeepLabv3+ BEiTv2 large Label Adaptation MIC Histogram Matching
Builds on: Split Shape Grammar CGA shape grammar Fake It Till You Make It MIC Label Adaptation
Closest papers by meaning
- What Makes Synthetic Data Effective in Image Segmentation (ICML 2026 · similarity 0.86)
- Separating Knowledge and Perception with Procedural Data (ICML 2025 · similarity 0.85)
- VLMaterial: Procedural Material Generation with Large Vision-Language Models (ICLR 2025 · similarity 0.84)
- Infusing Synthetic Data with Real-World Patterns for Zero-Shot Material State Segmentation (NeurIPS 2024 · similarity 0.83)
- InfiniBench: Infinite Benchmarking for Visual Spatial Reasoning with Customizable Scene Complexity (CVPR 2026 · similarity 0.83)
- MaskFactory: Towards High-quality Synthetic Data Generation for Dichotomous Image Segmentation (NeurIPS 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.