Research map / Point clouds / Point cloud classification
URDF-Anything: Constructing Articulated Objects with 3D Multimodal Language Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes URDF-Anything, an end-to-end 3D MLLM (ShapeLLM-7B with LoRA) that takes point clouds and text, autoregressively outputs URDF joint parameters and [SEG]-token-driven per-link point-cloud segmentation.
Problem
Automatically reconstructing articulated objects as functional URDF digital twins from visual observations, which previously required multi-stage pipelines with mesh libraries or separate segmentation stages.
Approach
Use a 3D multimodal LLM with a LISA-style [SEG] token whose hidden state (fused with the preceding category token) queries point-cloud features via cross-attention, so symbolic kinematic JSON and per-part geometric masks are generated jointly and trained end-to-end.
Key results
- Segmentation mIoU 0.63 overall vs 0.54 for best baseline Uni3D w/ text (16.7% relative gain); Count Acc 0.97 vs 0.84.
- Joint origin error 0.164 vs 0.207 for Articulate-Anything on all classes; axis error 0.132 vs 0.145.
- Physical executability 78% overall vs 52% for Articulate-Anything (50% relative gain).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use dynamic [SEG] tokens to couple segmentation with kinematic prediction rather than static text-guided segmentation | Uni3D w/ text segmentation baseline · PartNet-Mobility ALL and OOD | mIoU: 0.54 -> 0.63 (ALL); 0.51 -> 0.62 (OOD) | Helped |
| use dynamic [SEG] tokens to couple segmentation with kinematic prediction rather than static text-guided segmentation | Uni3D w/ text segmentation baseline · PartNet-Mobility ALL and OOD | Count Accuracy: 0.84 -> 0.97 (ALL); 0.82 -> 0.96 (OOD) | Helped |
| use URDF-Anything instead of Articulate-Anything | Articulate-Anything · PartNet-Mobility ALL | Type Error: \meanstd{0.025}{0.005} -> \textbf{\meanstd{0.008}{0.001}} | Helped |
| use URDF-Anything instead of Articulate-Anything | Articulate-Anything · PartNet-Mobility ALL | Axis Error: \meanstd{0.145}{0.450} -> \textbf{\meanstd{0.132}{0.048}} | Helped |
| use URDF-Anything instead of Articulate-Anything | Articulate-Anything · PartNet-Mobility ALL | Origin Error: \meanstd{0.207}{0.392} -> \textbf{\meanstd{0.164}{0.026}} | Helped |
| use URDF-Anything instead of Articulate-Anything | Articulate-Anything · PartNet-Mobility ALL | Physical executability: 52% -> 78% | Helped |
| replace point-cloud and text input with OBB representation and text (ablation) | URDF-Anything · PartNet-Mobility | Type Error: 0.008 -> \meanstd{0.42}{0.18} | Hurt |
| replace point-cloud and text input with OBB representation and text (ablation) | URDF-Anything · PartNet-Mobility | Axis Error: 0.132 -> \meanstd{0.70}{0.26} | Hurt |
| replace point-cloud and text input with OBB representation and text (ablation) | URDF-Anything · PartNet-Mobility | Origin Error: 0.164 -> \meanstd{0.47}{0.22} | Hurt |
| remove language guidance and use point cloud only (ablation) | URDF-Anything · PartNet-Mobility | Type Error: 0.008 -> \meanstd{0.34}{0.11} | Hurt |
| replace point-cloud input with image and text input, fine-tuning Qwen2.5-VL-7B (ablation) | URDF-Anything · PartNet-Mobility | Axis Error: 0.132 -> \meanstd{0.81}{0.24} | Hurt |
| remove [SEG] and segmentation loss from joint training (ablation) | URDF-Anything · PartNet-Mobility | Type Error: 0.008 -> 0.009 | Hurt |
Surprising findings
- Fine-tuned 2D image MLLM Qwen2.5-VL-7B yields much higher axis error than 3D-based approaches, suggesting explicit 3D geometry is needed.
- Segmentation-only training reduces mIoU and count accuracy relative to joint training, indicating kinematic structure learning helps segmentation.
Limitations
- Cannot generate physical properties such as mass or inertia.
- Not fully end-to-end: relies on an external point-to-mesh conversion module for link geometry.
- Numerical precision limited by token-based generation.
Experimental setup
Datasets: PartNet-Mobility PARIS real-world dataset
Benchmarks: PartNet-Mobility
Metrics: mIoU Count Accuracy joint type error joint axis error joint origin error physical executability rate Chamfer Distance
Baselines: Articulate-Anything Real2Code URDFormer Uni3D w/ text Uni3D w/o text Qwen2.5-VL-7B CARTO PARIS
Builds on: ShapeLLM LISA Uni3D DUSt3R LGM
Closest papers by meaning
- URDF-Anything+: End-to-End Generation for Simulation-Ready Articulated Assets (NeurIPS 2026 · similarity 0.91)
- Articulate-Anything: Automatic Modeling of Articulated Objects via a Vision-Language Foundation Model (ICLR 2025 · similarity 0.88)
- Real2Code: Reconstruct Articulated Objects via Code Generation (ICLR 2025 · similarity 0.86)
- AutoURDF: Unsupervised Robot Modeling from Point Cloud Frames Using Cluster Registration (CVPR 2025 · similarity 0.86)
- ManipLLM: Embodied Multimodal Large Language Model for Object-Centric Robotic Manipulation (CVPR 2024 · similarity 0.86)
- ArtFormer: Controllable Generation of Diverse 3D Articulated Objects (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.