Research map / Video / Video-language understanding
VideoCAD: A Dataset and Model for Learning Long‑Horizon 3D CAD UI Interactions from Video
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces VideoCAD, a 41K-video synthetic dataset of Onshape CAD UI interactions generated by a rule-based bot from DeepCAD models, plus the VideoCADFormer autoregressive transformer for behavior cloning and a VQA benchmark probing LLM 3D reasoning.
Problem
Existing UI-agent datasets focus on short, low-complexity web/mobile tasks and lack long-horizon, precision 3D CAD interactions captured as video with action annotations.
Approach
Automatically replay human-authored DeepCAD sketch-extrude sequences in Onshape through a hybrid Selenium/PyAutoGUI bot to produce large-scale, temporally aligned video-action pairs, and train a causal transformer decoder over ViT-encoded target and UI frames to predict low-level UI commands.
Key results
- VideoCADFormer reaches 98.08% command accuracy and 82.35% parameter accuracy, versus 96.25% and 78.72% for VPT, with mean perfectly predicted action rate of 87.54% vs 83.81%.
- On geometric fidelity, VideoCADFormer achieves 41.0% mean Chamfer success rate vs 28.5% for VPT, and 26.0% invalid rate vs 39.5%.
- On the VideoCAD VQA benchmark, LLMs underperform; e.g., GPT-4.1 reaches 47% on extrusion counting and 18% on depth-related questions, and all tested LLM UI agents fail to complete a CAD construction.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use VideoCADFormer instead of VPT | VideoCAD action prediction · VideoCAD test | command accuracy (%): 96.25 -> 98.08 | Helped |
| use VideoCADFormer instead of VPT | VideoCAD action prediction · VideoCAD test | parameter accuracy (%): 78.72 -> 82.35 | Helped |
| use VideoCADFormer instead of VPT | VideoCAD action prediction · VideoCAD test | mean perfectly predicted actions (%): 83.81 -> 87.54 | Helped |
| use VideoCADFormer instead of VPT | CAD generation · VideoCAD geometric fidelity, mean across sequence lengths | Chamfer success rate (%): 28.5 -> 41.0 | Helped |
| use VideoCADFormer instead of VPT | CAD generation · VideoCAD geometric fidelity, mean across sequence lengths | mean Chamfer distance: 0.0473 -> 0.0374 | Helped |
| use VideoCADFormer instead of VPT | CAD generation · VideoCAD geometric fidelity, mean across sequence lengths | invalid sample rate (%): 39.5 -> 26.0 | Helped |
| add past action inputs (ablation) | VideoCADFormer · VideoCAD ablation test | mean perfectly predicted actions (%): 73.11 -> 79.27 | Helped |
| add past UI state inputs (ablation) | VideoCADFormer · VideoCAD ablation test | mean perfectly predicted actions (%): 77.86 -> 79.27 | Helped |
| add past action and UI state inputs (ablation) | VideoCADFormer · VideoCAD ablation test | mean perfectly predicted actions (%): 5.59 -> 79.27 | Helped |
| add timestep embeddings (ablation) | VideoCADFormer · VideoCAD ablation test | mean perfectly predicted actions (%): 79.39 -> 79.27 | No clear effect |
| add color jitter (ablation) | VideoCADFormer · VideoCAD ablation test | mean perfectly predicted actions (%): 79.27 -> 79.24 | Hurt |
| use sketches instead of kernel-generated target images (ablation) | VideoCADFormer · VideoCAD ablation test | mean perfectly predicted actions (%): 79.27 -> 79.31 | No clear effect |
Surprising findings
- Frontier LLMs such as GPT-4.1, Claude-3.7 and Gemini-2.5 fail to complete any full CAD construction as UI agents.
- Sketch-style inputs slightly outperform kernel-rendered images, while realistic (ControlNet) images perform worst.
Limitations
- All trajectories are synthetic from a rule-based bot, lacking natural variability in timing, errors, and strategy.
- Covers only sketch-extrude operations on Onshape, omitting fillets, sweeps, lofts and assemblies.
- End-to-end geometric evaluation is run only on a small subset (200 sequences) due to rendering cost.
Experimental setup
Datasets: DeepCAD
Benchmarks: VideoCAD test split VideoCAD VQA
Metrics: command accuracy parameter accuracy perfectly predicted actions Chamfer distance Chamfer success rate invalid rate VQA accuracy
Baselines: VPT Pix2Act Pearce et al. Random Human Expert
Builds on: Video Pre-training (VPT) Pix2Act DeepCAD
Closest papers by meaning
- cadrille: Multi-modal CAD Reconstruction with Reinforcement Learning (ICLR 2026 · similarity 0.87)
- GeoCAD: Local Geometry-Controllable CAD Generation with Large Language Models (NeurIPS 2025 · similarity 0.86)
- CAD-GPT: Synthesising CAD Construction Sequence with Spatial Reasoning-Enhanced Multimodal LLMs (AAAI 2025 · similarity 0.85)
- FlexCAD: Unified and Versatile Controllable CAD Generation with Fine-tuned Large Language Models (ICLR 2025 · similarity 0.85)
- Text2CAD: Generating Sequential CAD Designs from Beginner-to-Expert Level Text Prompts (NeurIPS 2024 · similarity 0.85)
- CAD-Editor: A Locate-then-Infill Framework with Automated Training Data Synthesis for Text-Based CAD Editing (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.