Research map / Video / Video generation
AirSketch: Generative Motion to Sketch
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AirSketch, which trains a ControlNet on SDXL with augmentation-based self-supervision to map noisy hand-tracking images of air drawings into clean sketches, and releases two air-drawing datasets.
Problem
Generating clean, faithful, aesthetic sketches from highly noisy marker-less hand-tracking images of air drawing, without paired sketch-video training data and without AR/VR hardware.
Approach
Treat ControlNet's spatial conditioning as a soft constraint: apply sketch-specific augmentations (local, structural, false-stroke) to clean Quick, Draw! sketches to form distorted conditions, and train ControlNet to reconstruct the original clean sketch, so the adapter learns a noisy-to-clean mapping.
Key results
- With augmentation-based training on SDXL, the real-dataset SSIM rose from 0.55 (tracking) to 0.64 and Chamfer Distance fell from 31.99 to 25.46 relative to the no-augmentation baseline.
- The augmented ControlNet outperformed Sketch-Pix2Seq on SSIM and CD for the selected categories, while Sketch-Pix2Seq needs a separate model per category.
- Performance on 10 held-out categories was close to seen-category scores (synthetic SSIM 0.64, CD 17.27), indicating generalization.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add augmentation-based training | ControlNet with SDXL · Quick, Draw! seen and unseen categories, synthetic and real tracking | SSIM | Helped |
| Add augmentation-based training | ControlNet with SDXL · Quick, Draw! seen and unseen categories, synthetic and real tracking | CD | Helped |
| Use local augmentation only (ablation) | AirSketch trained on SD1.5 · Quick, Draw! seen categories | SSIM: 0.55 -> 0.60 | Helped |
| Use structural augmentation only (ablation) | AirSketch trained on SD1.5 · Quick, Draw! seen categories | SSIM: 0.55 -> 0.55 | No clear effect |
| Use false-stroke augmentation only (ablation) | AirSketch trained on SD1.5 · Quick, Draw! seen categories | CD: 32.36 -> 29.04 | Helped |
| Use structural augmentation only (ablation) | AirSketch trained on SD1.5 · Quick, Draw! seen categories | CD: 32.36 -> 31.14 | Helped |
| Use all three augmentation types (ablation) | AirSketch trained on SD1.5 · Quick, Draw! seen categories | CD: 32.36 -> 27.33 | Helped |
| Remove text prompt (ablation) | AirSketch · Quick, Draw! unseen categories | CLIP I2T: -24.7% | Hurt |
| Remove text prompt (ablation) | AirSketch · Quick, Draw! seen categories | CLIP I2T: -4.9% | Hurt |
| Increase input chaos (ablation) | AirSketch · Quick, Draw! tracking inputs | CD | Helped |
| Train the conditioning model with augmentation-based recovery | T2IAdapter · AirSketch sketch generation | Faithfulness | Mixed |
Surprising findings
- Spatial conditioning in ControlNet need not be a hard constraint; with augmentation it can map highly noisy inputs to clean outputs.
- Without augmentation, ControlNet fails to improve over the raw tracking image, and text prompts give little gain.
Limitations
- Only simple, cartoon-like sketches from Quick, Draw! are addressed; full-color images are not explored.
- The real air-drawing dataset is small (500 pairs), and the method relies on text prompts for unseen categories.
- Societal risks from misuse for generating mocking content.
Experimental setup
Datasets: Quick, Draw! TUBerlin
Benchmarks: Synthetic Air-Drawing dataset Real Air-Drawing dataset
Metrics: SSIM Chamfer Distance LPIPS CLIP image-to-image similarity CLIP image-to-text similarity
Baselines: Sketch-Pix2Seq ControlNet without augmentation Tracking images (no generation) SDXL without augmentation
Builds on: ControlNet T2IAdapter Stable Diffusion XL LoRA MediaPipe Hands
Closest papers by meaning
- FlipSketch: Flipping Static Drawings to Text-Guided Sketch Animations (CVPR 2025 · similarity 0.87)
- Breathing Life Into Sketches Using Text-to-Video Priors (CVPR 2024 · similarity 0.87)
- It's All About Your Sketch: Democratising Sketch Control in Diffusion Models (CVPR 2024 · similarity 0.86)
- StickMotion: Generating 3D Human Motions by Drawing a Stickman (CVPR 2025 · similarity 0.85)
- VLOGGER: Multimodal Diffusion for Embodied Avatar Synthesis (CVPR 2025 · similarity 0.85)
- LineArt: A Knowledge-guided Training-free High-quality Appearance Transfer for Design Drawing with Diffusion Model (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.