Research map / 3D vision / Face and shape
PyTorchGeoNodes: Enabling Differentiable Shape Programs for 3D Shape Reconstruction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Compiles Blender geometry-node shape programs into differentiable PyTorch code and fits their discrete and continuous parameters to RGB-D scans via a genetic algorithm with gradient refinement, plus procedural Gaussian splatting.
Problem
Recovering interpretable shape-program parameters (discrete and continuous) of furniture objects from RGB-D scans, where gradient descent cannot handle discrete parameters and prior learned predictors generalize poorly from synthetic to real data.
Approach
A compiler parses Blender geometry-node shape programs into differentiable PyTorch/PyTorch3D code, and a genetic algorithm searches discrete parameters while gradient descent refines continuous ones within each generation, with an objective of chamfer distance, occlusion penalty, and floor constraint.
Key results
- On ScanNet tables, the genetic algorithm reaches 0.14 m chamfer distance versus 0.16 for coordinate descent and 0.52 for GeoCode, and 0.08 rad absolute rotation error versus 0.12 and 0.45 respectively (before refinement, after-refinement numbers in Table 1 reported as 0.11 and 0.06).
- Discrete parameter accuracy for table top shape reaches 0.94 with the genetic algorithm, versus 0.78 for GeoCode and 0.72 for coordinate descent.
- The method uses no training data and generalizes to real ScanNet scenes where the supervised GeoCode baseline trained on synthetic data performs poorly.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use a genetic algorithm instead of Coordinate Descent | PyTorchGeoNodes shape-parameter search · ScanNet, all categories | Chamfer distance [m]: 0.16 -> 0.11 | Helped |
| use a genetic algorithm instead of Coordinate Descent | PyTorchGeoNodes shape-parameter search · ScanNet, all categories | Abs. rotation error [rad]: 0.11 -> 0.06 | Helped |
| use a genetic algorithm instead of GeoCode | ScanNet shape reconstruction · ScanNet, all categories | Chamfer distance [m]: 0.21 -> 0.11 | Helped |
| use a genetic algorithm instead of Coordinate Descent | Table shape-parameter reconstruction · ScanNet Table | Classification Accuracy, Top Shape: 0.72 -> 0.94 | Helped |
| use a genetic algorithm instead of Coordinate Descent | Table shape-parameter reconstruction · ScanNet Table | Classification Accuracy, Legs Type: 0.6 -> 0.78 | Helped |
| use a genetic algorithm instead of Coordinate Descent | Table shape-parameter reconstruction · ScanNet Table | Classification Accuracy, Has Cabinet Leg: 0.55 -> 0.86 | Helped |
| use a genetic algorithm instead of Coordinate Descent | Table shape-parameter reconstruction · ScanNet Table | Classification Accuracy, Has Mid Board: 0.49 -> 0.76 | Helped |
| add gradient-based refinement (ablation) | Genetic algorithm · ScanNet, all categories | Chamfer distance [m]: 0.14 -> 0.11 | Helped |
| add gradient-based refinement (ablation) | Genetic algorithm · ScanNet, all categories | Abs. rotation error [rad]: 0.08 -> 0.06 | Helped |
| add gradient-based refinement (ablation) | GeoCode · ScanNet, all categories | Chamfer distance [m]: 0.52 -> 0.21 | Helped |
| add gradient-based refinement (ablation) | GeoCode · ScanNet, all categories | Abs. rotation error [rad]: 0.45 -> 0.12 | Helped |
Surprising findings
- The supervised GeoCode method trained on synthetic data generalizes poorly to ScanNet, with the domain gap large enough that the optimization-based coordinate descent baseline outperforms it substantially.
Limitations
- Shape programs must be designed by hand for each object category, limiting expressiveness in the wild.
- Method struggles to recover parameters that define very small details.
- Limited to rigid objects; deformable and articulated objects and textures are not handled.
Experimental setup
Datasets: ScanNet Scannotate
Benchmarks: ScanNet
Metrics: mean absolute difference for continuous parameters classification accuracy for discrete parameters chamfer distance absolute rotation error
Baselines: GeoCode coordinate descent
Builds on: GeoCode Scannotate Gaussian splatting PyTorch3D
Closest papers by meaning
- Learning to Infer and Execute 3D Shape Programs (ICLR 2019 · similarity 0.88)
- Unsupervised 3D Shape Reconstruction by Part Retrieval and Assembly (CVPR 2023 · similarity 0.87)
- MeshCoder: LLM-Powered Structured Mesh Code Generation from Point Clouds (NeurIPS 2025 · similarity 0.86)
- From Points to Multi-Object 3D Reconstruction (CVPR 2021 · similarity 0.86)
- Learning Partonomic 3D Reconstruction from Image Collections (CVPR 2025 · similarity 0.86)
- Enhancing Implicit Shape Generators Using Topological Regularizations (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.