Research map / Large language models / Agents, code and math
SketchAgent: Language-Driven Sequential Sketch Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SketchAgent, which uses a frozen off-the-shelf multimodal LLM (mainly Claude 3.5 Sonnet) with in-context examples of a string-based sketching language on a numbered grid to draw sketches stroke by stroke, fitting Bezier curves to the output, with no training.
Problem
Existing sketch generation methods either optimize all strokes simultaneously (losing the sequential drawing process) or use LLM-generated SVG/code that looks mechanical, and they lack support for interactive, collaborative sketching.
Approach
Treat a multimodal LLM as a sketching agent: it outputs stroke sequences as grid-cell coordinates plus t-values, which are fit to cubic Bezier curves via least squares and rendered onto a numbered canvas, enabling stroke-by-stroke generation, chat-based editing, and human-agent collaboration without any training.
Key results
- SketchAgent with Claude 3.5 Sonnet reaches Top-1 0.23 / Top-5 0.43 CLIP recognition on 500 QuickDraw-category sketches (ablation), with the 'Claude3.5-Sonnet (SVG)' baseline at 0.27 Top-1 and human sketches at 0.27 Top-1 in Table 1 (default setting reported as 0.27 Top-1 in the main table).
- In a 2AFC study with 150 participants, SketchAgent sketches were chosen as human-drawn 74.90% of the time against direct SVG prompting, versus 38.9% for direct prompting when compared with human sketches.
- Chat-based editing followed instructions correctly 92% of the time over 54 sketches.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the system prompt (ablation) | SketchAgent · QuickDraw 50 categories | Top-1 accuracy: 0.20 -> 0.23 | Helped |
| Add the system prompt (ablation) | SketchAgent · QuickDraw 50 categories | Top-5 accuracy: 0.42 -> 0.43 | Helped |
| Include chain-of-thought instructions and thinking tags (ablation) | SketchAgent · QuickDraw 50 categories | Top-1 accuracy: 0.14 -> 0.23 | Helped |
| Include chain-of-thought instructions and thinking tags (ablation) | SketchAgent · QuickDraw 50 categories | Top-5 accuracy: 0.29 -> 0.43 | Helped |
| Provide a complete sketch example instead of a single-stroke example (ablation) | SketchAgent · QuickDraw 50 categories | Top-1 accuracy: 0.07 -> 0.23 | Helped |
| Provide a complete sketch example instead of a single-stroke example (ablation) | SketchAgent · QuickDraw 50 categories | Top-5 accuracy: 0.16 -> 0.43 | Helped |
| Use a 10 × 10 grid instead of 50 × 50 (ablation) | SketchAgent · QuickDraw 50 categories | Top-1 accuracy: 0.23 -> 0.14 | Hurt |
| Use a 10 × 10 grid instead of 50 × 50 (ablation) | SketchAgent · QuickDraw 50 categories | Top-5 accuracy: 0.43 -> 0.28 | Hurt |
| Use a 25 × 25 grid instead of 50 × 50 (ablation) | SketchAgent · QuickDraw 50 categories | Top-1 accuracy: 0.23 -> 0.19 | Hurt |
| Use a 25 × 25 grid instead of 50 × 50 (ablation) | SketchAgent · QuickDraw 50 categories | Top-5 accuracy: 0.43 -> 0.42 | No clear effect |
| Use a 75 × 75 grid instead of 50 × 50 (ablation) | SketchAgent · QuickDraw 50 categories | Top-1 accuracy: 0.23 -> 0.23 | No clear effect |
| Use a 75 × 75 grid instead of 50 × 50 (ablation) | SketchAgent · QuickDraw 50 categories | Top-5 accuracy: 0.43 -> 0.41 | No clear effect |
Surprising findings
- Direct SVG prompting achieves recognition scores comparable to SketchAgent, yet its outputs look mechanical and are less human-like.
- Providing a complete sketch example in the prompt matters more than removing other components; a single-stroke example drops Top-1 to 0.07.
Limitations
- Constrained by the text-oriented priors of the backbone LLM, so recognizability of complex objects and human figures is low.
- Struggles with drawing letters and numbers.
- Recognition performance drops as concept complexity (number of letters) increases.
Experimental setup
Datasets: QuickDraw
Benchmarks: QuickDraw
Metrics: Top-1 accuracy Top-5 accuracy 2AFC human-likeness preference stroke count distribution CLIPScore chat editing instruction-following accuracy
Baselines: GPT-4o GPT-4o-mini Claude3-Opus Claude3.5-Sonnet (SVG prompting) SVGDreamer SketchRNN QuickDraw human sketches
Builds on: chain-of-thought prompting in-context learning SketchRNN CLIP zero-shot classification
Closest papers by meaning
- DialogDraw: Image Generation and Editing System Based on Multi-Turn Dialogue (AAAI 2025 · similarity 0.89)
- Sketchtopia: A Dataset and Foundational Agents for Benchmarking Asynchronous Multimodal Communication with Iconic Feedback (CVPR 2025 · similarity 0.88)
- Text to Sketch Generation with Multi-Styles (NeurIPS 2025 · similarity 0.88)
- Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Models (NeurIPS 2024 · similarity 0.88)
- It's All About Your Sketch: Democratising Sketch Control in Diffusion Models (CVPR 2024 · similarity 0.86)
- SketchThinker-R1: Towards Efficient Sketch-Style Reasoning in Large Multimodal Models (ICLR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.