Research map / 3D vision / 3D generation
Generative Perception of Shape and Material from Differential Motion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains a from-scratch conditional video diffusion model (U-ViT3D-Mixer) in pixel space that jointly generates shape (normals) and material maps from short differential-motion videos, yielding multimodal samples for static inputs that converge with motion.
Problem
Recovering object shape and material from a single image is inherently ambiguous under unknown lighting; existing learning methods predict one attribute at a time and a single "best" output, ignoring ambiguity and not exploiting motion cues.
Approach
A conditional denoising diffusion model trained from scratch on synthetic three-frame object-motion videos, with local spatio-temporal neighborhood attention at high resolution, global 3D attention at low resolution, and a channel mixer that jointly denoises normals, albedo, roughness, metallic, and specularity, so that samples capture ambiguity for static inputs and concentrate as motion is obser
Key results
- On DiLiGenT shape estimation, the model reaches 18.03 mean AE and 12.31 median AE, competitive with specialized shape models (Marigold-E2E-FT 17.74 mean AE), and on MIT Intrinsic albedo it reaches SSIM 0.80 and PSNR 23.14, beating ColorfulShading (0.72, 20.56).
- On the synthetic differential-motion set, adding local spatio-temporal attention and the multi-attribute mixer lowers Mean AE with motion from 18.757 (base) to 11.262 with 100.3M parameters.
- Inference takes about 2.7 s per three-frame video on a single A100 with 2.8 GB GPU memory, using 50 DDIM steps.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add local spatio-temporal attention modules (ablation) | U-ViT3D · Differential-motion evaluation set, with motion | Mean AE: 18.757 -> 14.555 | Helped |
| add local spatio-temporal attention modules (ablation) | U-ViT3D · Differential-motion evaluation set, with motion | SSIM: 0.778 -> 0.825 | Helped |
| add lightweight channel mixers to local 3D blocks (ablation) | + Local Spatio-Temporal Attention · Differential-motion evaluation set, with motion | Mean AE: 14.555 -> 11.262 | Helped |
| add lightweight channel mixers to local 3D blocks (ablation) | + Local Spatio-Temporal Attention · Differential-motion evaluation set, with motion | SSIM: 0.825 -> 0.832 | Helped |
| add local spatio-temporal attention modules (ablation) | U-ViT3D · Differential-motion evaluation set, without motion | Mean AE: 22.471 -> 18.957 | Helped |
| add local spatio-temporal attention modules (ablation) | U-ViT3D · Differential-motion evaluation set, without motion | SSIM: 0.760 -> 0.811 | Helped |
| add lightweight channel mixers to local 3D blocks (ablation) | + Local Spatio-Temporal Attention · Differential-motion evaluation set, without motion | Mean AE: 18.957 -> 16.886 | Helped |
| add lightweight channel mixers to local 3D blocks (ablation) | + Local Spatio-Temporal Attention · Differential-motion evaluation set, without motion | SSIM: 0.811 -> 0.823 | Helped |
| provide moving-object observations instead of static observations (ablation) | U-ViT3D-Mixer · Differential-motion evaluation set | Mean AE: 16.886 -> 11.262 | Helped |
| provide moving-object observations instead of static observations (ablation) | U-ViT3D-Mixer · Differential-motion evaluation set | SSIM: 0.823 -> 0.832 | Helped |
| use best 3 of 10 samples instead of mean prediction (ablation) | IID · MIT-Intrinsic | SSIM: 0.61 -> 0.44 | Hurt |
| use best 3 of 10 samples instead of mean prediction (ablation) | IID · MIT-Intrinsic | RMSE: 0.13 -> 0.15 | Hurt |
Surprising findings
- Due to training on textured rather than homogeneous objects, the model occasionally produces a 'postcard' flat interpretation for ambiguous static images.
- A model trained only to predict albedo yields plausible surface normals when probed with DPT-style probes, suggesting cross-attribute representation sharing.
Limitations
- Restricted to rigid, horizontal-style object motions and specific materials; does not handle deforming objects, multiple distinct motions, or translucent/transparent objects.
- Requires objects to be pre-segmented from background.
- Trained only on synthetic data, with out-of-distribution generalization shown qualitatively and on limited real-world examples.
Experimental setup
Datasets: Adobe 3D Assets (1100 models) 400 open-source texture assets 200 HDR environment maps DiLiGenT MIT Intrinsic Image Dataset Stanford-ORB
Benchmarks: DiLiGenT MIT Intrinsic Image Dataset Stanford-ORB held-out synthetic differential-motion evaluation set
Metrics: Mean AE Median AE SSIM RMSE PSNR LPIPS
Baselines: Marigold-E2E-FT StableNormal DSINE GeoWizard IID ColorfulShading Marigold--Albedo RGB-X
Builds on: DDPM DDIM U-ViT Hourglass Diffusion Transformer Disney principled BSDF
Closest papers by meaning
- Multistable Shape from Shading Emerges from Patch Diffusion (NeurIPS 2024 · similarity 0.89)
- Denoising Functional Maps: Diffusion Models for Shape Correspondence (CVPR 2025 · similarity 0.87)
- Towards Physical Understanding in Video Generation: A 3D Point Regularization Approach (NeurIPS 2025 · similarity 0.87)
- Diff-2-in-1: Bridging Generation and Dense Perception with Diffusion Models (ICLR 2025 · similarity 0.87)
- Decompositional Neural Scene Reconstruction with Generative Diffusion Prior (CVPR 2025 · similarity 0.86)
- Intrinsic Image Diffusion for Indoor Single-view Material Estimation (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.