Research map / 3D vision / 3D generation
Generating Images with 3D Annotations Using Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes 3D-DST, which conditions ControlNet on edge maps rendered from ShapeNet/Objaverse CAD models plus LLM-generated text prompts, to create diffusion-generated training images with automatic 3D annotations for classification and 3D pose estimation.
Problem
Diffusion models lack explicit 3D geometry control, so generated images cannot be given controlled poses/distances or automatic ground-truth 3D annotations, limiting their use for augmenting 3D-aware and OOD training data.
Approach
Render CAD models from sampled viewpoints and distances, extract Canny edge maps as ControlNet visual prompts, and diversify appearance with LLM-generated text prompts (backgrounds, colors, weather), so the rendering parameters give free 3D labels for the generated images.
Key results
- On ImageNet-100 with DeiT-B, 3D-DST pre-training improves top-1 accuracy by 3.83 points over the baseline (84.28 to 88.11).
- On ImageNet-R (OOD) with DeiT-Ti, accuracy rises from 49.96 to 56.12 (+6.16), outperforming Text2Img augmentation (+2.62).
- On PASCAL3D+ with NeMo, 3D-DST improves Acc@pi/18 from 57.12 to 62.51 ID and OOD-CV Acc@pi/6 from 55.31 to 58.81.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| pretrain on 3D-DST synthetic data instead of target data alone | DeiT-S · ImageNet-100 ID | accuracy: 84.56 -> 88.96 | Helped |
| pretrain on 3D-DST synthetic data instead of target data alone | DeiT-S · ImageNet-R OOD | accuracy: 49.61 -> 56.65 | Helped |
| pretrain on 3D-DST synthetic data | DeiT-S · ImageNet-200 | accuracy: 81.50 -> 84.81 | Helped |
| pretrain on 3D-DST synthetic data | ResNet · PASCAL3D+ and ObjectNet3D ID | Acc@π/18: +3.92 percentage points | Helped |
| pretrain on 3D-DST synthetic data | NeMo · PASCAL3D+ and ObjectNet3D ID | Acc@π/18: +5.39 percentage points | Helped |
| pretrain on 3D-DST synthetic data | NeMo · OOD-CV | Acc@π/18: 26.57 -> 26.44 | Hurt |
| add LLM-generated text prompts to 3D-DST (ablation) | DeiT-S · ImageNet-100 ID | accuracy: 87.82 -> 88.96 | Helped |
| add LLM-generated text prompts to 3D-DST (ablation) | DeiT-S · ImageNet-R OOD | accuracy: 55.95 -> 56.65 | Helped |
| replace rendered 3D edge-map conditioning with ImageNet image edges (ablation) | DeiT-S image augmentation · ImageNet-R OOD | accuracy: 49.61 -> 48.69 | Hurt |
| use rendered objects with random 3D environment backgrounds (ablation) | DeiT-S image augmentation · ImageNet-R OOD | accuracy: 49.61 -> 54.81 | Helped |
| add camera augmentation to DST-3D pretraining (ablation) | CubeRCNN · ARKitScenes | AP3D: 42.74 -> 43.19 | Helped |
| apply K-fold consistency filtering to remove noisy annotations (ablation) | DST-3D bus data · bus synthetic data | PCS: 59.5 -> 66.0 | Helped |
Surprising findings
- Text2Img pre-training without 3D control yields mixed results, sometimes lowering ID accuracy (e.g., DeiT-B drops 0.18 points).
- Around 25% of generated images were judged inconsistent with the prompts by human evaluators, with failures concentrated on uncommon viewpoints such as cars seen from below.
Limitations
- Uncommon viewpoints (e.g., cars from below) produce failure images with noisy 3D annotations.
- About 25% of generated images are not consistent with the textual and visual prompts according to human evaluation.
- Relies on CAD models from repositories with limited diversity in shape and texture.
Experimental setup
Datasets: ImageNet-100 ImageNet-200 ImageNet-R ShapeNet Objaverse PASCAL3D+ ObjectNet3D OOD-CV ARKitScenes BG-20k
Benchmarks: ImageNet-100 ImageNet-200 ImageNet-R PASCAL3D+ ObjectNet3D OOD-CV ARKitScenes
Metrics: top-1 accuracy Acc@pi/6 Acc@pi/18 AP2D AP3D real-to-synthetic performance pose consistency score
Baselines: Text2Img diffusion augmentation AugMix NeMo CubeRCNN
Builds on: ControlNet Stable Diffusion NeMo DeiT
Closest papers by meaning
- DIRECT-3D: Learning Direct Text-to-3D Generation on Massive Noisy 3D Data (CVPR 2024 · similarity 0.90)
- DiffusionGAN3D: Boosting Text-guided 3D Generation and Domain Adaptation by Combining 3D GANs and Diffusion Priors (CVPR 2024 · similarity 0.89)
- PI3D: Efficient Text-to-3D Generation with Pseudo-Image Diffusion (CVPR 2024 · similarity 0.89)
- Sculpt3D: Multi-View Consistent Text-to-3D Generation with Sparse 3D Prior (CVPR 2024 · similarity 0.89)
- Geometry Image Diffusion: Fast and Data-Efficient Text-to-3D with Image-Based Surface Representation (ICLR 2025 · similarity 0.89)
- MVDream: Multi-view Diffusion for 3D Generation (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.