Research map / Vision-language and multimodal / Human and embodied modeling
Text-to-CAD Generation Through Infusing Visual Feedback in Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CADFusion, which fine-tunes LLaMA-3-8B on text-to-CAD parametric sequences and alternates with DPO-based visual feedback using LVM-scored rendered CAD images.
Problem
Text-to-CAD generation from text descriptions into CAD parametric sequences has relied only on sequential (ground-truth sequence) supervision, ignoring visual feedback from rendered objects, which is non-differentiable and many-to-one.
Approach
Alternate between supervised sequential learning on ground-truth parametric sequences and DPO-based visual feedback, where preference pairs are built by scoring rendered CAD images with a large vision-language model (LLaVA-OneVision) instead of humans.
Key results
- CADFusion achieves LVM score 8.96 and invalidity ratio 6.20%, versus GPT-4o 5.13 LVM score with 74.26% IR and Text2CAD 2.01 LVM score.
- CADFusion sketch F1 85.22 and extrusion F1 92.79, with Chamfer Distance 19.89 versus Text2CAD 30.23 (expert prompt).
- Ablation: removing visual feedback (SL only) lowers LVM score from 8.96 to 7.69 while reducing IR from 6.20 to 4.84.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add visual feedback to sequential learning (ablation) | CADFusion · Test set | LVM Score: 7.69 -> 8.96 | Helped |
| Add visual feedback to sequential learning (ablation) | CADFusion · Test set | IR: 4.84 -> 6.20 | Hurt |
| Add human refinement of sequential-learning captions (ablation) | CADFusion SL · Test set | LVM Score: 6.56 -> 7.69 | Helped |
| Add human refinement of sequential-learning captions (ablation) | CADFusion SL · Test set | IR: 6.00 -> 4.84 | Helped |
| Alternate visual-feedback and sequential-learning updates instead of training with VF only (ablation) | CADFusion after initial SL · Test set | IR: 88.87 -> 6.20 | Helped |
| Alternate visual-feedback and sequential-learning updates instead of training with VF only (ablation) | CADFusion after initial SL · Test set | LVM Score: 5.94 -> 8.96 | Helped |
| Replace DPO with RPO stabilization (ablation) | CADFusion SL-VF · Test set | LVM Score: 5.94 -> 6.21 | No clear effect |
| Use LVM-scored rather than human-scored preference pairs (ablation) | CADFusion SL-VFSL(1) · Test set | LVM Score: 8.28 -> 8.76 | Helped |
| Use LVM-scored rather than human-scored preference pairs (ablation) | CADFusion SL-VFSL(1) · Test set | IR: 17.03 -> 4.42 | Helped |
| Increase alternating rounds from one to three (ablation) | CADFusion · Test set | LVM Score: 8.76 -> 8.89 | Helped |
| Increase alternating rounds from three to five (ablation) | CADFusion · Test set | LVM Score: 8.89 -> 8.96 | Helped |
| Increase alternating rounds from three to five (ablation) | CADFusion · Test set | IR: 4.21 -> 6.20 | Hurt |
Surprising findings
- Extended visual-feedback-only training drove invalidity ratio to 88.87%, so alternating with sequential learning is necessary.
- Text2CAD performance dropped sharply when given the authors' simplified prompts, indicating sensitivity to prompt detail.
Limitations
- LVMs degrade when given multiple images, so feedback uses a single view only.
- Struggles with very complex shapes and letter/word shapes requiring spatial and commonsense reasoning.
- Discrepancies arise when prompts involve more than about 8 distinct items.
Experimental setup
Datasets: DeepCAD (via SkexGen processing) CADFusion text-sequence dataset (20k, human-refined captions) LVM-scored preference pairs
Benchmarks: DeepCAD-derived test split (5%)
Metrics: F1 Sketch F1 Extrusion Chamfer Distance Coverage Minimum Matching Distance Jensen-Shannon Divergence Invalidity Ratio LVM Score Avg. Rank
Baselines: GPT-4o (8-shot) Text2CAD
Builds on: direct preference optimization Text2CAD SkexGen DeepCAD
Closest papers by meaning
- CAD-Llama: Leveraging Large Language Models for Computer-Aided Design Parametric 3D Model Generation (CVPR 2025 · similarity 0.90)
- From 2D CAD Drawings to 3D Parametric Models: A Vision-Language Approach (AAAI 2025 · similarity 0.90)
- Generating CAD Code with Vision-Language Models for 3D Designs (ICLR 2025 · similarity 0.89)
- CAD-Coder: Text-to-CAD Generation with Chain-of-Thought and Geometric Reward (NeurIPS 2025 · similarity 0.89)
- Text2CAD: Generating Sequential CAD Designs from Beginner-to-Expert Level Text Prompts (NeurIPS 2024 · similarity 0.88)
- cadrille: Multi-modal CAD Reconstruction with Reinforcement Learning (ICLR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.