Research map / Video / Video generation
Stable Part Diffusion 4D: Multi-View RGB and Kinematic Parts Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Stable Part Diffusion 4D (SP4D) extends the SV4D 2.0 multi-view video diffusion model with a parallel part-segmentation branch that jointly generates RGB and kinematic part videos from monocular inputs, using spatial color encoding, bidirectional diffusion fusion, and a contrastive part consistency loss.
Problem
Generating kinematic (articulation-aligned) part decompositions that are consistent across views and time from monocular video or single-image inputs, where existing auto-rigging and appearance-based part segmentation methods generalize poorly.
Approach
A dual-branch video diffusion model where the part branch encodes kinematic part masks as RGB-like images via deterministic spatial color encoding (3D part-center coordinates), so it can share the RGB VAE, with bidirectional fusion between branches and a contrastive loss enforcing part identity consistency across views and frames.
Key results
- On KinematicParts20K val (multi-view), SP4D reaches mIoU 0.68 and F1 0.70 versus 0.15 mIoU for SAM2 and 0.17 for DeepViT.
- User study average rating of 4.26 for SP4D versus 1.96 (SAM2) and 1.85 (DeepViT) on a 1-5 scale.
- Rigging precision of 72.7 on KinematicParts20K-test versus 64.3 for UniRig and 63.7 for Magic Articulate.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the part consistency loss (ablation) | SP4D · KinematicParts20K val, multi-view | mIoU: 0.38 -> 0.68 | Helped |
| add the part consistency loss (ablation) | SP4D · KinematicParts20K val, multi-view | ARI: 0.15 -> 0.60 | Helped |
| add the part consistency loss (ablation) | SP4D · KinematicParts20K val, multi-frame | mIoU: 0.44 -> 0.70 | Helped |
| add the part consistency loss (ablation) | SP4D · KinematicParts20K val, multi-frame | ARI: 0.22 -> 0.63 | Helped |
| add BiDiFuse (ablation) | SP4D · KinematicParts20K val, multi-view | mIoU: 0.57 -> 0.68 | Helped |
| add BiDiFuse (ablation) | SP4D · KinematicParts20K val, multi-view | ARI: 0.51 -> 0.60 | Helped |
| add BiDiFuse (ablation) | SP4D · KinematicParts20K val, multi-frame | mIoU: 0.61 -> 0.70 | Helped |
| add BiDiFuse (ablation) | SP4D · KinematicParts20K val, multi-frame | ARI: 0.58 -> 0.63 | Helped |
| use SP4D instead of Segment Any Mesh | KinematicParts20K-test 3D segmentation · KinematicParts20K-test | mIoU: 0.15 -> 0.64 | Helped |
| use SP4D instead of Magic Articulate | KinematicParts20K-test rigging · KinematicParts20K-test | precision: 63.7 -> 72.7 | Helped |
| use SP4D instead of Magic Articulate | rigging generated objects · Generated Objects | User Study: 2.7 -> 4.1 | Helped |
Surprising findings
- SAM2 variant prompted with ground-truth part centroids (SAM2*) still performs poorly, suggesting appearance cues fundamentally misalign with kinematic boundaries.
Limitations
- Camera parameterization limited to azimuth and elevation with a simple lens model; struggles with strong perspective or complex trajectories.
- Trained primarily assuming a single object per scene; multi-object scenes are challenging.
- Code and data release not committed.
Experimental setup
Datasets: ObjaverseDy KinematicParts20K Objaverse XL ObjaverseDy++ DAVIS
Benchmarks: KinematicParts20K validation set KinematicParts20K test set DAVIS
Metrics: mIoU ARI F1 mAcc user study rating rigging precision
Baselines: SAM2 SAM2* DeepViT Segment Any Mesh SAMPart3D Magic Articulate UniRig
Builds on: SV4D 2.0 Stable Video Diffusion SV3D EDM
Closest papers by meaning
- Diffusion4D: Fast Spatial-temporal Consistent 4D generation via Video Diffusion Models (NeurIPS 2024 · similarity 0.88)
- PartGen: Part-level 3D Generation and Reconstruction with Multi-view Diffusion Models (CVPR 2025 · similarity 0.88)
- PartCrafter: Structured 3D Mesh Generation via Compositional Latent Diffusion Transformers (NeurIPS 2025 · similarity 0.88)
- Motion4D: Learning 3D-Consistent Motion and Semantics for 4D Scene Understanding (NeurIPS 2025 · similarity 0.87)
- CAT4D: Create Anything in 4D with Multi-View Video Diffusion Models (CVPR 2025 · similarity 0.87)
- 4Diffusion: Multi-view Video Diffusion Model for 4D Generation (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.