Research map / 3D vision / 3D generation
SKDream: Controllable Multi-view and 3D Generation with Arbitrary Skeletons
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Generalizes skeleton-conditioned generation to arbitrary (non-human) skeletons by building a 24k mesh-skeleton dataset (Objaverse-SK), a skeleton extraction pipeline, and a multi-view diffusion model with coordinate color encoding and a skeletal correlation module, followed by LRM reconstruction and UV texture refinement.
Problem
Skeletal conditioned controllable generation has been limited to human skeletons; there is no large-scale paired data of arbitrary object meshes and skeletons, and 2D projected skeletons are ambiguous for complex 3D structures.
Approach
Encode 3D skeleton joints and bones as coordinate-colored 2D condition images (CCE with inverse-depth alpha), inject them via a Skeletal Correlation Module (self-attention for anatomy, cross-view attention and AdaLN camera embedding) into MVDream, then reconstruct with InstantMesh and refine texture in UV space.
Key results
- Ours-SCM achieves mean instance SKA score 80.43 vs 70.13 for SDEdit and 74.69 for Ours-Raw (Table 1).
- Ours-SCM achieves 33.28% win rate in PickScore vs 19.56% for SDEdit (Table 2).
- Skeleton extraction pipeline reaches ~80% success rate vs ~15% for RigNet.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SCM instead of the raw condition pipeline (ablation) | SKDream · Objaverse-SK evaluation set | SKA score (MeanInst.): 74.69 -> 80.43 | Helped |
| Add depth alpha to coordinate color encoding (CCE-D) (ablation) | SCM-based SKDream · Objaverse-SK evaluation set | SKA score (Animals): 90.45 -> 91.16 | Helped |
| Add depth alpha to coordinate color encoding (CCE-D) (ablation) | SCM-based SKDream · Objaverse-SK evaluation set | SKA score (Humans): 74.82 -> 78.45 | Helped |
| Add depth alpha to coordinate color encoding (CCE-D) (ablation) | SCM-based SKDream · Objaverse-SK evaluation set | SKA score (Plants): 52.57 -> 53.53 | Helped |
| Use SCM instead of convolutional blocks (ablation) | CCE-D SKDream · Objaverse-SK evaluation set | SKA score (Animals): 88.53 -> 91.16 | Helped |
| Use SCM instead of convolutional blocks (ablation) | CCE-D SKDream · Objaverse-SK evaluation set | SKA score (Humans): 75.04 -> 78.45 | Helped |
| Use SCM instead of convolutional blocks (ablation) | CCE-D SKDream · Objaverse-SK evaluation set | SKA score (Plants): 46.17 -> 53.53 | Helped |
| Use CCE-D instead of binary skeleton encoding (ablation) | SCM-based SKDream · Objaverse-SK evaluation set | SKA score (Animals): 89.20 -> 91.16 | Helped |
| Use CCE-D instead of binary skeleton encoding (ablation) | SCM-based SKDream · Objaverse-SK evaluation set | SKA score (Humans): 76.04 -> 78.45 | Helped |
| Use CCE-D instead of binary skeleton encoding (ablation) | SCM-based SKDream · Objaverse-SK evaluation set | SKA score (Plants): 46.16 -> 53.53 | Helped |
| Use SCM with layer normalization rather than convolutional blocks (ablation) | SKDream condition encoder · Training convergence | convergence rate: 4× faster | Helped |
| Use SCM with adaptive layer normalization rather than layer normalization (ablation) | SKDream condition encoder · Training convergence | training steps to SKA score 70: 1k | Helped |
Surprising findings
- The learning-based RigNet skeletonization was found unsatisfactory, producing mostly symmetric skeletons with about 15% success rate.
Limitations
- Skeletons have limited descriptive ability for non-tree-structured objects.
- Only global skeletons without fine-grained part semantics are considered.
- Reconstructed textures are blurry without the refinement stage; artifacts appear in uncovered UV regions.
Experimental setup
Datasets: Objaverse Objaverse-SK ModelResources ShapeNet
Benchmarks: Objaverse-SK evaluation set (108 skeleton instances) ShapeNet (generalization)
Metrics: SKA score PickScore CLIP score win rate skeleton success rate
Baselines: SDEdit SDEdit+COSAG RigNet Ours-Raw
Builds on: MVDream ControlNet InstantMesh Mean Curvature Flow DINOv2 SDEdit
Closest papers by meaning
- ArtFormer: Controllable Generation of Diverse 3D Articulated Objects (CVPR 2025 · similarity 0.86)
- Template-free Articulated Gaussian Splatting for Real-time Reposable Dynamic View Synthesis (NeurIPS 2024 · similarity 0.86)
- Joint2Human: High-Quality 3D Human Generation via Compact Spherical Embedding of 3D Joints (CVPR 2024 · similarity 0.85)
- Learning Implicit Representation for Reconstructing Articulated Objects (ICLR 2024 · similarity 0.85)
- SyncHuman: Synchronizing 2D and 3D Generative Models for Single-view Human Reconstruction (NeurIPS 2025 · similarity 0.85)
- RigGS: Rigging of 3D Gaussians for Modeling Articulated Objects in Videos (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.