Research map / Vision-language and multimodal / CLIP and zero-shot
Open Vocabulary Semantic Scene Sketch Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Fine-tunes a CLIP ViT-B/16 sketch encoder with v-v attention, two-level hierarchical triplet training, and cross-attention on FS-COCO sketch-caption pairs to achieve open-vocabulary scene sketch segmentation without pixel labels.
Problem
Machine understanding of abstract freehand scene sketches at the pixel level is underexplored and lacks pixel-level annotated data; existing methods need synthetic or fully supervised data.
Approach
A two-level hierarchical CLIP ViT encoder (holistic scene-level and category-level triplet losses sharing weights) with v-v self-attention in a second path, learnable visual prompts, LayerNorm-only tuning, and text-to-sketch cross-attention in later layers, trained only from sketch-caption pairs.
Key results
- Reaches 85.54% pixel accuracy (Acc@P) and 73.48 mIoU on the FS-COCO test set, versus 59.98 mIoU for the strongest fine-tuned baseline CLIP Surgery.
- Outperforms zero-shot CLIP segmentation by about 37 points in pixel accuracy as claimed in the abstract (17.33 mIoU for zero-shot CLIP).
- On the Ge et al. freehand sketches, achieves 53.94 mIoU vs 37.16 for LDP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add v-v attention to the visual encoder (ablation) | Ours · FS-COCO test | mIoU: 43.55 -> 73.48 | Helped |
| add the category-level network stage (ablation) | Ours · FS-COCO test | mIoU: 65.03 -> 73.48 | Helped |
| use learnable thresholding instead of cosine-similarity pixel weighting (ablation) | Ours · FS-COCO test | mIoU: 66.93 -> 73.48 | Helped |
| add the global scene loss (ablation) | Ours · FS-COCO test | mIoU: 69.06 -> 73.48 | Helped |
| add text-vision cross-attention (ablation) | Ours · FS-COCO test | mIoU: 70.13 -> 73.48 | Helped |
| use category-level triplet-loss features from multiple layers instead of only the last layer (ablation) | Ours · FS-COCO test | mIoU: 71.29 -> 73.48 | Helped |
| combine holistic and category-level sketch training with v-v attention and learnable thresholding | CLIP Surgery⋆⋆ · FS-COCO test | mIoU: 59.98 -> 73.48 | Helped |
| train on FS-COCO sketch-caption pairs with the proposed method instead of LDP trained on semi-synthetic sketches | Scene sketch segmentation · Ge et al. freehand sketches | mIoU: 37.16 -> 53.94 | Helped |
Surprising findings
- On sketches with correlated categories, the model performs similarly to CLIP Surgery, indicating a limitation of language supervision.
- On strong-performance sketches the model nearly matches human annotators (0.11-point gap), but falls well short on ambiguous and numerous-category sketches.
Limitations
- Performance drops on ambiguous sketches, sketches with interchangeable categories, and sketches with more than six categories.
- Correlated categories remain a limitation of language supervision.
- Domain gap on symbolic-style sketches from Ge et al. dataset.
Experimental setup
Datasets: FS-COCO MS-COCO
Benchmarks: FS-COCO Ge et al. freehand sketch dataset
Metrics: mIoU Acc@P Acc@S
Baselines: CLIP GroupViT SegCLIP CLIP Surgery LDP ZegCLIP DenseCLIP Grounded-SAM
Builds on: CLIP CLIP Surgery visual prompt tuning
Closest papers by meaning
- O3SLM: Open Weight, Open Data, and Open Vocabulary Sketch-Language Model (AAAI 2026 · similarity 0.89)
- Learning Dense Correspondences between Photos and Sketches (ICML 2023 · similarity 0.89)
- SceneTrilogy: On Human Scene-Sketch and Its Complementarity With Photo and Text (CVPR 2023 · similarity 0.89)
- Enhance Sketch Recognition’s Explainability via Semantic Component-Level Parsing (AAAI 2024 · similarity 0.88)
- SketchFusion: Learning Universal Sketch Features through Fusing Foundation Models (CVPR 2025 · similarity 0.88)
- Lexicon3D: Probing Visual Foundation Models for Complex 3D Scene Understanding (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.