Research map / Vision-language and multimodal / Multimodal LLMs

O3SLM: Open Weight, Open Data, and Open Vocabulary Sketch-Language Model

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Builds SketchVCL, a large image-sketch-instruction dataset generated by an automatic sketch pipeline, and trains O3SLM (LLaVA-1.5-based 7B/13B LVLMs) for sketch-based counting, detection, SBIR, and VQA, outperforming open LVLMs.

Problem

Large vision-language models struggle to understand hand-drawn sketches as queries for grounding, counting, retrieval and reasoning, largely due to the lack of large-scale datasets jointly pairing sketches, photos and instructions.

Approach

Generate instance-level sketches for large detection datasets (OpenImages, Objects365, COCO) via a Pix2Pix-plus-edge pipeline, create image-sketch-instruction triplets for a two-stage pretraining and instruction tuning, and train an LVLM that concatenates sketch, image and text tokens, with SBIR cast as yes/no token prediction.

Key results

• O3SLM-7B reaches 43.5 average counting accuracy on PixMo-Count versus 33.6 for GPT-4o, the best closed baseline.
• On Sketchy SBIR, O3SLM-7B achieves Acc@1 of 65.0 compared with 11.0 for LLaVA-1.5-7B.
• O3SLM-13B reaches 44.0 counting accuracy on PixMo-Count averaged over sketch sources, with the largest gap on the unseen TU Berlin sketches (47.0 vs 34.4 for Pixtral-12B).

Surprising findings

• Pretraining has minimal effect on counting, though it strongly helps SBIR.
• Image-only performance drops slightly (VQAv2 80.0 to 76.6 for 13B) while text-only object detection improves markedly relative to LLaVA.

Limitations

• Struggles with many highly overlapping objects, sometimes producing extra bounding boxes.
• Relies on synthetic Pix2Pix-based sketches for pretraining rather than human-drawn sketches.

Experimental setup

Datasets: SketchVCL Objects365 OpenImages COCO PixMo-Count Sketchy QuickDraw! TU Berlin ShareGPT4V

Benchmarks: PixMo-Count COCO Sketchy QuickDraw! TU Berlin VQAv2 MME LLaVA-in-the-Wild SeedBench

Metrics: accuracy Acc@K mAP

Baselines: GPT-4o GPT-4o-mini Gemini 1.5 Pro Gemini 1.5 Flash LLaVA-1.5 Qwen2.5-VL-7B DeepSeek-VL2 Molmo-7B-D Pixtral-12B OneVision

Builds on: LLaVA-1.5 Molmo Pix2Pix SAM2

Closest papers by meaning

• Open Vocabulary Semantic Scene Sketch Understanding (CVPR 2024 · similarity 0.89)
• Language-Image Models with 3D Understanding (ICLR 2025 · similarity 0.88)
• Enhancing Large Vision Language Models with Self-Training on Image Comprehension (NeurIPS 2024 · similarity 0.87)
• Zebra-CoT: A Dataset for Interleaved Vision-Language Reasoning (ICLR 2026 · similarity 0.87)
• GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Models (ICLR 2026 · similarity 0.87)
• Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Models (NeurIPS 2024 · similarity 0.87)