Research map / 3D vision / 3D generation
Neural Assets: 3D-Aware Multi-Object Scene Synthesis with Image Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Neural Assets are learnable per-object appearance and 3D-box pose tokens, obtained from paired video frames, that replace text tokens to fine-tune Stable Diffusion for multi-object 3D-aware editing and cross-scene composition.
Problem
Controlling the 3D pose and placement of multiple objects in a scene for image diffusion models, which existing 2D or single-object 3D-aware methods cannot handle on real-world multi-object scenes due to scarce paired 3D training data.
Approach
Represent each object as a Neural Asset whose appearance tokens are pooled (RoIAlign over DINO features) from a source frame while its pose tokens come from 3D box corners in a different target frame; training on video frame pairs forces disentangled appearance and pose, and the tokens replace text tokens in a fine-tuned text-to-image diffusion model.
Key results
- On OBJect translation (unseen objects), Neural Assets reach 20.13 PSNR and 0.274 LPIPS versus 15.20 PSNR and 0.477 LPIPS for 3DIT.
- On multi-object MOVi-E, object-level PSNR is 20.05 versus 12.41 for 3DIT and 15.06 for Chained.
- On Objectron ablations, DINO ViT-B/8 with joint fine-tuning (FT-DINO) gives the best object-level PSNR of 16.41 versus 14.13 for CLIP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Neural Assets instead of 3DIT | OBJect unseen Translation · OBJect unseen Translation | PSNR: 15.20 -> 20.13 | Helped |
| Use Neural Assets instead of 3DIT | OBJect unseen Rotation · OBJect unseen Rotation | PSNR: 16.28 -> 18.39 | Helped |
| Use Neural Assets instead of 3DIT | OBJect unseen Removal · OBJect unseen Removal | PSNR: 24.66 -> 28.44 | Helped |
| Use Neural Assets instead of the strongest baseline | multi-object editing · MOVi-E, Objectron, and Waymo Open object-level | PSNR: 15.06 -> 20.05; 11.30 -> 16.41; 16.20 -> 17.67 | Helped |
| Use a fine-tuned DINO encoder instead of CLIP (ablation) | Neural Assets · Objectron image-level | PSNR: 12.95 -> 14.83 | Helped |
| Use a fine-tuned DINO encoder instead of MAE (ablation) | Neural Assets · Objectron image-level | PSNR: 13.64 -> 14.83 | Helped |
| Fine-tune DINO rather than keeping it frozen (ablation) | Neural Assets · Objectron image-level | PSNR: 13.75 -> 14.83 | Helped |
| Add background modeling (ablation) | Neural Assets · Objectron image-level | PSNR: 12.98 -> 14.83 | Helped |
| Add relative camera pose to background tokens (ablation) | Neural Assets · Objectron image-level | PSNR: 13.71 -> 14.83 | Helped |
| Train on source-target frame pairs rather than a single frame (ablation) | Neural Assets · Objectron image-level | PSNR: 12.63 -> 14.83 | Helped |
| Train on source-target frame pairs rather than single frames without positional encoding (ablation) | Neural Assets · Objectron image-level | PSNR: 13.74 -> 14.83 | Helped |
| Remove positional encoding during single-frame training (ablation) | Neural Assets · Objectron image-level | PSNR: 12.63 -> 13.74 | Helped |
Surprising findings
- Training on single frames with positional encoding in the appearance encoder leads the model to ignore the pose token, since the appearance token already encodes object position.
- Using relative camera pose as background pose token yields notably better object-level metrics, not only image-level ones.
Limitations
- Symmetry ambiguity causes flipped object orientation (e.g., cup handle) under 180-degree rotation.
- Camera-object motion entanglement: background moves when translating the foreground object on Objectron.
- Requires 3D bounding box annotations, limiting training to datasets that provide them; cannot control deformable or articulated changes.
Experimental setup
Datasets: OBJect MOVi-E Objectron Waymo Open
Benchmarks: OBJect MOVi-E Objectron Waymo Open
Metrics: PSNR SSIM LPIPS FID DINO feature similarity
Baselines: 3DIT Chained (SAM + SD inpainting + Zero-1-to-3)
Builds on: Stable Diffusion 3DIT Zero-1-to-3 RoIAlign DINO
Closest papers by meaning
- DORSal: Diffusion for Object-centric Representations of Scenes $\textit{et al.}$ (ICLR 2024 · similarity 0.89)
- Generating Images with 3D Annotations Using Diffusion Models (ICLR 2024 · similarity 0.88)
- AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformers (CVPR 2025 · similarity 0.88)
- VD3D: Taming Large Video Diffusion Transformers for 3D Camera Control (ICLR 2025 · similarity 0.87)
- Controlling Space and Time with Diffusion Models (ICLR 2025 · similarity 0.86)
- Object Pose Estimation via the Aggregation of Diffusion Features (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.