Research map / 3D vision / Face and shape
Real2Code: Reconstruct Articulated Objects via Code Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Real2Code reconstructs multi-part articulated objects by segmenting parts with a fine-tuned SAM, completing part shapes with a learned occupancy network, and having a LoRA-fine-tuned CodeLlama-7B predict joints as Python/MJCF code from oriented bounding boxes.
Problem
Reconstructing articulated objects (geometry and joint parameters) from visual observations, which prior methods handle only for objects with one or two moving parts and which generalize poorly to real-world objects.
Approach
Formulate joint prediction as code generation by a fine-tuned code LLM that consumes oriented bounding boxes of reconstructed parts and selects OBB axes and edges (classification rather than regression), outputting simulation-ready Python MJCF code.
Key results
- Real2Code achieves joint rotation error 0.63 and position error 0.07 on 4-5 part objects with type accuracy 0.97, while Ditto and PARIS fail on these objects.
- Real2Code can reconstruct objects with up to 15 parts (6-15 part group) despite training on objects with up to 7 parts; rotation error 30.2 and type accuracy 0.89 on this group.
- Fine-tuned SAM reaches 92% mean IoU on used masks and 84% match rate to ground-truth masks on unseen objects.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace selecting OBB edges with absolute continuous joint-position regression (ablation) | Real2Code articulation prediction · 6-15 parts | position error: 0.03 -> 0.7 | Hurt |
| replace selecting OBB edges with position regression relative to the OBB center (ablation) | Real2Code articulation prediction · 6-15 parts | position error: 0.03 -> 0.06 | Hurt |
| add RGB image input alongside OBBs (ablation) | Real2Code articulation prediction · 6-15 parts | position error: 0.03 -> 0.02 | No clear effect |
| use predicted OBBs instead of oracle ground-truth bounding boxes (ablation) | Real2Code articulation prediction · 6-15 parts | rotation error: 0.0 -> 30.2 | Hurt |
| use Real2Code instead of Ditto | articulated object reconstruction · Box (2 parts) | whole CD: 1.73 -> 1.31 | Helped |
| use Real2Code instead of Ditto | articulated object reconstruction · Furniture (2-4 parts) | whole CD: 2.25 -> 3.47 | Hurt |
| fine-tune SAM to align segmentation with object kinematics | SAM part segmentation · unseen object images | mean IoU | Helped |
| use the part-level shape completion model rather than extracting meshes from segmented point clouds | Real2Code reconstruction · PartNet-Mobility test objects | part CD: 7.19 -> 3.02 | Helped |
Surprising findings
- Adding RGB image input to the LLM yields no clear improvement over OBB-only input.
- Zero-shot GPT-4 gets reasonable joint positions on multi-part objects, but its rotation errors are large (e.g., 57.3 on 2-part objects).
Limitations
- Handles only a single object per scene; multi-object scenes need detection and preprocessing.
- Predicts only joint type, position, and axis; joint limits and friction need extra interactions.
- Articulation accuracy is sensitive to segmentation failures, which corrupt OBBs.
Experimental setup
Datasets: PartNet-Mobility PartNet
Benchmarks: PartNet-Mobility test split (35 objects)
Metrics: Chamfer distance rotation error position error joint type accuracy IoU match rate
Baselines: PARIS Ditto GPT-4 Real2Code-SegOnly
Builds on: Segment Anything (SAM) Convolutional Occupancy Networks CodeLlama DUSt3R Ditto PARIS
Closest papers by meaning
- Learning Partonomic 3D Reconstruction from Image Collections (CVPR 2025 · similarity 0.87)
- SINGAPO: Single Image Controlled Generation of Articulated Parts in Objects (ICLR 2025 · similarity 0.87)
- ArtFormer: Controllable Generation of Diverse 3D Articulated Objects (CVPR 2025 · similarity 0.87)
- REACTO: Reconstructing Articulated Objects from a Single Video (CVPR 2024 · similarity 0.87)
- DIPO: Dual-State Images Controlled Articulated Object Generation Powered by Diverse Data (NeurIPS 2025 · similarity 0.87)
- Articulate your NeRF: Unsupervised articulated object modeling via conditional view synthesis (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.