Research map / Vision-language and multimodal / Human and embodied modeling

CADTalk: An Algorithm and Benchmark for Semantic Commenting of CAD Programs

CVPR 2024 · Vision-language and multimodal › Human and embodied modeling · approach: Abstract visual reasoning benchmarks · 4 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces semantic commenting of CAD (OpenSCAD) programs and the CADTalk benchmark, and proposes CADTalker, which executes programs, renders photorealistic multiview images via ControlNet, segments parts with Grounding DINO and SAM, and votes labels back to code blocks.

Problem

CAD programs encode shapes as code that is hard to understand and edit without semantic comments; the task is to segment programs into code blocks and label each with the semantic shape part it produces.

Approach

Move the commenting problem from the program domain to the image domain: execute the program, render multiview depth maps, convert them to photorealistic images with ControlNet, use open-vocabulary detection and segmentation (Grounding DINO + SAM) to find parts, then aggregate confidence votes across views back onto code blocks via pixel-to-block masks.

Key results

• CADTalker reaches 88.75% block accuracy on CADTalk-Cube (GPT words) and 78.29% block accuracy on CADTalk-Real.
• Compared with PartSLIP, CADTalker achieves 88.75% vs 29.67% average block accuracy on CADTalk-Cube.
• Ablations show removing ControlNet translation drops average block accuracy from 88.75% to 45.20% on CADTalk-Cube.

Measured effects reported in the paper

Surprising findings

• Removing the photorealistic ControlNet translation step causes a large accuracy drop, showing that textureless renderings are poorly recognized by vision models.
• LLM few-shot commenting of CAD programs succeeds only for programs similar to the example and fails to generalize to new shapes or primitives.

Experimental setup

Datasets: PartNet CADTalk

Benchmarks: CADTalk-Cube CADTalk-Ellip CADTalk-Real

Metrics: block accuracy semantic IoU

Baselines: PartSLIP PartSLIP++ GPT-4 few-shot commenting

Builds on: ControlNet Grounding DINO Segment Anything Model PartSLIP

Closest papers by meaning

• From 2D CAD Drawings to 3D Parametric Models: A Vision-Language Approach (AAAI 2025 · similarity 0.88)
• ShapeWalk: Compositional Shape Editing Through Language-Guided Chains (CVPR 2024 · similarity 0.86)
• cadrille: Multi-modal CAD Reconstruction with Reinforcement Learning (ICLR 2026 · similarity 0.86)
• Generating CAD Code with Vision-Language Models for 3D Designs (ICLR 2025 · similarity 0.86)
• CAD-Llama: Leveraging Large Language Models for Computer-Aided Design Parametric 3D Model Generation (CVPR 2025 · similarity 0.86)
• CADCrafter: Generating Computer-Aided Design Models from Unconstrained Images (CVPR 2025 · similarity 0.85)