Research map / 3D vision / 3D generation
CAD-GPT: Synthesising CAD Construction Sequence with Spatial Reasoning-Enhanced Multimodal LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds CAD-GPT, a LLaVA-1.5-7B-based MLLM that generates CAD construction sequences from a single image or text by adding discretized 3D orientation, 3D position, and 2D sketch tokens with learnable position embeddings.
Problem
MLLMs such as GPT-4 struggle with 3D spatial reasoning (sketch plane origin, orientation, extrusion direction) and have high failure rates when generating CAD construction sequences from images or text.
Approach
A 3D Modeling Spatial Localization Mechanism that unfolds 3D sketch-plane rotation angles and origin positions into discrete special tokens added to the LLM vocabulary, plus discretized 2D sketch coordinate tokens and learnable position embeddings for these token types.
Key results
- Image input: CAD-GPT median CD 9.77 vs HNC-CAD 18.64 and GPT-4 62.64; invalidity ratio 1.61 vs 18.64 for HNC-CAD.
- Text input: CAD-GPT invalidity ratio 7.43 and median CD 28.33 vs GPT-4 76.97 IR and 187.52 CD.
- Ablation: removing the localization tokens raised image-input IR from 1.61 to 37.15 and median CD from 9.77 to 161.31.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the 3D Modeling Spatial Localization Mechanism (ablation) | CAD-GPT · Image input | IR: 37.15 -> 1.61 | Helped |
| add the 3D Modeling Spatial Localization Mechanism (ablation) | CAD-GPT · Image input | Median CD: 161.31 -> 9.77 | Helped |
| add learnable position embeddings (ablation) | CAD-GPT with localization tokens · Image input | IR: 4.31 -> 1.61 | Helped |
| add learnable position embeddings (ablation) | CAD-GPT with localization tokens · Image input | Median CD: 27.98 -> 9.77 | Helped |
| add the 3D Modeling Spatial Localization Mechanism (ablation) | CAD-GPT · Text input | IR: 40.23 -> 7.43 | Helped |
| add the 3D Modeling Spatial Localization Mechanism (ablation) | CAD-GPT · Text input | Median CD: 145.87 -> 28.33 | Helped |
| add learnable position embeddings (ablation) | CAD-GPT with localization tokens · Text input | IR: 10.12 -> 7.43 | Helped |
| add learnable position embeddings (ablation) | CAD-GPT with localization tokens · Text input | Median CD: 29.58 -> 28.33 | Helped |
Surprising findings
- GPT-4 scored higher than DeepCAD/SkexGen/HNC-CAD on command accuracy yet had far higher invalidity ratio (64.37) and CD.
Limitations
- Relies on single fixed-view rendered images and text descriptions rather than richer 3D inputs.
- Text-description subset is small (18k) and was manually curated, and many DeepCAD models lack natural category descriptions.
Experimental setup
Datasets: DeepCAD OpenCascade-rendered images (160k+) GPT-4o-filtered CAD captions (18k)
Benchmarks: DeepCAD held-out set
Metrics: Chamfer Distance Invalidity Ratio ACC_cmd ACC_param
Baselines: DeepCAD SkexGen HNC-CAD GPT-4 LLaMA-3.1 Qwen2-VL-Max
Builds on: LLaVA-1.5 DeepCAD MeshGPT PolyGen SkexGen
Closest papers by meaning
- CAD-Llama: Leveraging Large Language Models for Computer-Aided Design Parametric 3D Model Generation (CVPR 2025 · similarity 0.89)
- CAD-Editor: A Locate-then-Infill Framework with Automated Training Data Synthesis for Text-Based CAD Editing (ICML 2025 · similarity 0.88)
- cadrille: Multi-modal CAD Reconstruction with Reinforcement Learning (ICLR 2026 · similarity 0.88)
- From 2D CAD Drawings to 3D Parametric Models: A Vision-Language Approach (AAAI 2025 · similarity 0.88)
- GeoCAD: Local Geometry-Controllable CAD Generation with Large Language Models (NeurIPS 2025 · similarity 0.87)
- CAD-SIGNet: CAD Language Inference from Point Clouds using Layer-wise Sketch Instance Guided Attention (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.