Research map / Detection and segmentation / Open-set and few-shot detection
Infusing Synthetic Data with Real-World Patterns for Zero-Shot Material State Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds a synthetic data pipeline that maps texture patterns extracted unsupervisedly from real images into 2D/3D scenes, plus a 1,220-image point-annotated zero-shot material state segmentation benchmark (MatSeg), and trains a ConvNext U-Net that beats SAM and Materialistic.
Problem
Segmenting material states (wet, dry, infected, cooked, burned, etc.) across open-world domains is poorly covered by existing datasets; manual polygon annotation is costly and synthetic data lacks real-world diversity.
Approach
Extract random image channel maps and uniform textures from unlabeled real photos with simple heuristics, then use them as blending maps and PBR material properties to generate synthetic 2D/3D scenes, training a net with a per-pixel embedding and cosine-similarity softmax loss; evaluate with point-based triplet similarity annotations.
Key results
- On the MatSeg benchmark, the MatSeg-trained net reaches 0.91-0.92 triplet accuracy (all) versus 0.76 for Materialistic and 0.69-0.74 for SAM with 1-8 points.
- On the Cu and FeM mineral datasets, the MatSeg net achieves IoU 0.52 and 0.62 versus 0.36-0.37 for SAM at 1 point.
- SAM and Materialistic outperform the MatSeg net on Materialistic general (0.87 and 0.75 vs 0.75 MatSeg at 1 point) and on LabPics Chemistry, i.e., bulk-shaped materials.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train on a combination of 2D and 3D scenes rather than only 2D scenes (ablation) | MatSeg · MatSeg (All) | triplet accuracy: 0.91 -> 0.92 | Helped |
| Train on a combination of 2D and 3D scenes rather than only 3D scenes (ablation) | MatSeg · MatSeg (All) | triplet accuracy: 0.90 -> 0.92 | Helped |
| Train on a combination of 2D and 3D scenes rather than only 2D scenes (ablation) | MatSeg · MatSeg (Soft) | triplet accuracy: 0.84 -> 0.84 | No clear effect |
| Train on a combination of 2D and 3D scenes rather than only 3D scenes (ablation) | MatSeg · MatSeg (Soft) | triplet accuracy: 0.83 -> 0.84 | Helped |
| Use MatSeg-trained segmentation instead of Materialistic | zero-shot segmentation · MatSeg benchmark | triplet accuracy: 0.76 -> 0.92 | Helped |
| Use MatSeg-trained segmentation instead of SAM with 8 points | zero-shot segmentation · MatSeg benchmark | triplet accuracy: 0.74 -> 0.92 | Helped |
| Use MatSeg-trained segmentation instead of Materialistic | zero-shot segmentation · MatSeg benchmark (Soft) | triplet accuracy: 0.72 -> 0.84 | Helped |
| Use MatSeg-trained segmentation instead of SAM with 8 points | zero-shot segmentation · MatSeg benchmark (Soft) | triplet accuracy: 0.65 -> 0.84 | Helped |
| Use MatSeg instead of the strongest listed baseline | zero-shot segmentation · Corrosao, leaf disease, NuInsSeg, and CryoNuSeg benchmarks | IoU: 0.56 -> 0.69; 0.54 -> 0.56; 0.32 -> 0.38; 0.28 -> 0.45 | Helped |
| Use MatSeg instead of the strongest listed baseline | zero-shot segmentation · Cu, FeM, URDE, Soil-type-class-2, Soil type, Materialistic, LabPics Chemistry, and LabPics Medical | IoU: 0.52 -> 0.69; 0.62 -> 0.67; 0.50 -> 0.52; 0.62 -> 0.72; 0.69 -> 0.88; 0.75 -> 0.87; 0.62 -> 0.75; 0.71 -> 0.72 | Hurt |
Surprising findings
- SAM with 8 input points still underperforms the MatSeg net with a single point on the benchmark, suggesting SAM's training data lacks such material-state patterns.
- The MatSeg net scores below SAM and Materialistic on the Materialistic benchmark and some bulk-material datasets, indicating a trade-off in domain coverage.
Limitations
- Texture extraction assumes uniform regions and fails on bent surfaces or non-uniform illumination, so patterns are biased toward common textures.
- Lower accuracy on microscopy and tissue domains, attributed to a lack of such images in the source data.
- Benchmark annotation is heavily reliant on human judgment with considerable uncertainty, verified by only a second annotator.
Experimental setup
Datasets: Open Images v7 SA-1B (Segment Anything dataset) ShapeNet HDRI Haven Materialistic URDE LabPics CryoNuSeg NuInsSeg Corrosao Leaf diseases Soil type datasets
Benchmarks: MatSeg benchmark Materialistic Cu dataset FeM dataset corrosao-segment Leaf diseases URDE Soil-type-class-2 NuInsSeg CryoNuSeg LabPics Chemistry LabPics Medical
Metrics: triplet accuracy IoU at optimal threshold
Baselines: SAM Materialistic
Builds on: Materialistic Segment Anything ConvNeXt
Closest papers by meaning
- Material Palette: Extraction of Materials from a Single Image (CVPR 2024 · similarity 0.86)
- Intrinsic Image Diffusion for Indoor Single-view Material Estimation (CVPR 2024 · similarity 0.84)
- MaRI: Material Retrieval Integration across Domains (CVPR 2025 · similarity 0.83)
- Alchemist: Parametric Control of Material Properties with Diffusion Models (CVPR 2024 · similarity 0.83)
- WinSyn: : A High Resolution Testbed for Synthetic Data (CVPR 2024 · similarity 0.83)
- IDArb: Intrinsic Decomposition for Arbitrary Number of Input Views and Illuminations (ICLR 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.