Research map / 3D vision / 3D generation
BLURD: Benchmarking and Learning using a Unified Rendering and Diffusion Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces BLURD, which combines Blender path-traced 3D renders with Stable Diffusion 1.5 ControlNet conditioning to create paired synthetic-to-photorealistic face datasets with controllable factors, and uses them to benchmark CLIP backbones.
Problem
Pre-trained vision models such as CLIP are hard to evaluate for biases and factor-level behavior: synthetic 3D renders lack realism, while real-world photo datasets are costly and hard to control.
Approach
Render a controllable 3D human scene in Blender and convert it to photorealistic images with Stable Diffusion 1.5 conditioned on depth, normal, color palette ControlNets and the render via img2img, so that each photorealistic image keeps identical factors to its 3D counterpart.
Key results
- Human raters rated BLURD SD: Unleashed at 4.19 and BLURD SD: Harbinger at 3.98 on photo-realism versus 2.42 for BLURD 3D (CelebA-HQ 4.70).
- Zero-shot accuracy on single factors is consistently higher on BLURD SD: Harbinger than on BLURD 3D across CLIP backbones, indicating a 3D-to-photoreal gap.
- Segmentation U-Net trained on BLURD SD achieved higher IoU than BLURD 3D-trained models on CelebAMask-HQ under all three augmentation regimes.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace 3D renders with photo-realistic BLURD SD: Harbinger images | BLURD · Human photo-realism evaluation | perceived photo-realism: 2.42 -> 3.98 | Helped |
| remove depth, normal, image-to-image, and spatial-palette conditioning (ablation) | BLURD SD · Human photo-realism evaluation | perceived photo-realism: 3.98 -> 4.19 | Helped |
| weaken conditioning to prompt, depth map, and normal map (BLURD SD: Tempered) (ablation) | BLURD SD · Single-factor zero-shot classification | accuracy | Mixed |
| weaken conditioning to prompt only (BLURD SD: Unleashed) (ablation) | BLURD SD · Single-factor zero-shot classification | accuracy | Mixed |
| weaken conditioning to prompt, depth map, and normal map (BLURD SD: Tempered) (ablation) | BLURD SD · Multi-factor zero-shot classification | accuracy | Helped |
| weaken conditioning to prompt only (BLURD SD: Unleashed) (ablation) | BLURD SD · Multi-factor zero-shot classification | accuracy | Helped |
| use photo-realistic BLURD SD images instead of 3D renders for training | U-Net segmentation model · CelebAMask-HQ test set | IOU | Helped |
| replace high-quality assets with low-poly meshes and simple textures (ablation) | BLURD SD pipeline · Low-poly asset generation | controllability | Hurt |
| increase denoise level or modify ControlNet weights when using low-poly assets (ablation) | BLURD SD pipeline · Low-poly asset generation | controllability | Hurt |
Surprising findings
- 3D-to-photorealism equivariance was negatively correlated with ImageNet Top-1 accuracy for ViT-B-32 across training datasets, suggesting more training data can be detrimental to this symmetry.
- All races were often confused with Caucasian in the 3D renders, and CLIP models showed color preferences (OpenAI for green, LAION for violet).
Limitations
- Equivariance analysis is limited by the expense of training large CLIP models, giving small sample sizes.
- Pipeline is highly sensitive to low-poly geometry, producing artifacts that require reducing ControlNet weights or raising denoise, which trades off control.
- Results are not intended as the sole determinant of model bias or safety.
Experimental setup
Datasets: BLURD 3D BLURD SD BLURD Mask CelebAMask-HQ CelebA-HQ ImageNet
Benchmarks: BLURD 3D BLURD SD BLURD Mask CelebAMask-HQ ImageNet
Metrics: zero-shot accuracy multi-factor accuracy equivariance (cosine similarity) IoU human photo-realism rating (1-5)
Baselines: 3DShapes MPI3D PUG CelebA-HQ
Builds on: Stable Diffusion ControlNet CLIP Blender Cycles
Closest papers by meaning
- Diffusion Renderer: Neural Inverse and Forward Rendering with Video Diffusion Models (CVPR 2025 · similarity 0.89)
- BlurDM: A Blur Diffusion Model for Image Deblurring (NeurIPS 2025 · similarity 0.88)
- ImageNet-D: Benchmarking Neural Network Robustness on Diffusion Synthetic Object (CVPR 2024 · similarity 0.88)
- Uni-Renderer: Unifying Rendering and Inverse Rendering Via Dual Stream Diffusion (CVPR 2025 · similarity 0.88)
- Coherent 3D Scene Diffusion From a Single RGB Image (NeurIPS 2024 · similarity 0.88)
- Warm Diffusion: Recipe for Blur-Noise Mixture Diffusion Models (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.