Research map / Detection and segmentation / CNN features
ContextSeg: Sketch Semantic Segmentation by Querying the Context with Attention
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
ContextSeg is a two-stage sketch semantic segmentation method that learns stroke embeddings with an autoencoder plus dense distance-field regression, then labels groups of strokes auto-regressively with a Transformer decoder using pointer-style selection and focal loss.
Problem
Sketch semantic segmentation: assigning pre-defined part labels to strokes, where prior image-, sequence- and graph-based methods struggle to jointly capture stroke structure, positional information, and inter-stroke proximity.
Approach
Treat each whole stroke as a raster-image entity embedded by a coordinate-augmented CNN autoencoder with an extra narrow-band distance-field decoder, then decode groups of strokes per step with an auto-regressive Transformer so previously labeled and remaining strokes serve as context.
Key results
- On SPG, average component accuracy 92.6% vs 91.2% for Sketch-Segformer, 42.5%-level baselines lower; SAcc average 96.2% vs 95.2% (Sketch-Segformer).
- On CreativeSketch, average CAcc 55.8% vs 43.9% for Sketch-Segformer (about 11.9-point gain).
- Semantic-aware copy-paste augmentation raised eye/mouth stroke accuracy in ants from 69.3% to 89.1%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ContextSeg instead of Sketch-Segformer | Sketch semantic segmentation · SPG average | CAcc: 91.2 -> 92.6 | Helped |
| Use ContextSeg instead of Sketch-Segformer | Sketch semantic segmentation · CreativeSketch average | CAcc: 43.9 -> 55.8 | Helped |
| Replace the stroke embedding network with Sketchformer (ablation) | ContextSeg · SPG selected-category average | CAcc: 93.6 -> 68.6 | Hurt |
| Remove coordinate channels from the embedding network (ablation) | ContextSeg · SPG selected-category average | CAcc: 93.6 -> 86.3 | Hurt |
| Add distance-field regression to the embedding network (ablation) | ContextSeg · SPG selected-category average | CAcc: 90.9 -> 93.6 | Helped |
| Replace parallel decoding with autoregressive decoding (ablation) | ContextSeg · SPG selected-category average | CAcc: 91.6 -> 93.6 | Helped |
| Train jointly across the five selected categories | ContextSeg · SPG cross-category training | SAcc: +13.1; -0.3; +0.4; +0.2; +0.1 | Mixed |
| Apply semantic-aware copy-and-paste augmentation | ContextSeg · Ant | SAcc: 69.3 -> 89.1 | Helped |
| Apply semantic-aware copy-and-paste augmentation | ContextSeg · Ambulance | SAcc: 58.6 -> 76.2 | Helped |
Surprising findings
- Parallel decoding (Ours-PD) degrades CAcc noticeably (93.6% to 91.6% average on the ablation subset), showing auto-regressive group decoding matters.
- Cross-category training degraded some parts (e.g., crab face and airplane tail wings), indicating similarity of parts must be positional/geometric, not only semantic.
Limitations
- Embedding network cannot encode strokes with rapid and large variations.
- Segmentation Transformer fails to label highly cluttered strokes such as hairs and eyelashes.
- Lower accuracy on the face category due to overlapping strokes.
Experimental setup
Datasets: SPG CreativeSketch
Benchmarks: SPG CreativeSketch
Metrics: stroke accuracy (SAcc) grouping accuracy (GAcc) component accuracy (CAcc)
Baselines: SPGSeg SketchGNN Sketch-Segformer Sketchformer-Seg parallel decoding variant (Ours-PD)
Builds on: Transformer Pointer Networks CoordConv Sketchformer focal loss
Closest papers by meaning
- Open Vocabulary Semantic Scene Sketch Understanding (CVPR 2024 · similarity 0.84)
- SceneTrilogy: On Human Scene-Sketch and Its Complementarity With Photo and Text (CVPR 2023 · similarity 0.83)
- SketchFusion: Learning Universal Sketch Features through Fusing Foundation Models (CVPR 2025 · similarity 0.81)
- Leveraging Hallucinations to Reduce Manual Prompt Dependency in Promptable Segmentation (NeurIPS 2024 · similarity 0.81)
- Explore In-Context Segmentation via Latent Diffusion Models (AAAI 2025 · similarity 0.81)
- Integrating Low-Level Visual Cues for Enhanced Unsupervised Semantic Segmentation (AAAI 2025 · similarity 0.80)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.