Research map / Detection and segmentation / 3D detection and driving
Believing is Seeing: Unobserved Object Detection using Generative Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces unobserved object detection in 2D, 2.5D and 3D and adapts 2D diffusion (SDXL outpainting), 3D diffusion (DFM) and VLMs to predict spatio-semantic distributions of occluded or out-of-frame objects, benchmarked on RealEstate10k and NYU Depth V2.
Problem
Detecting objects that are near the camera but not visible in an image, either occluded or outside the image frame, which standard object detection cannot address.
Approach
Treat generative models as samplers of plausible scenes conditioned on one RGB image, run an object detector on the samples, and aggregate detections into a spatio-semantic distribution over 2D, 2.5D and 3D domains, with new metrics for evaluating against a single ground-truth realization.
Key results
- On the 2D task, SDXL achieves the best non-oracle cross-entropy (1.257 on RealEstate10k) and NYU accuracy 0.944, while DFM is competitive.
- DFM is the only evaluated method that predicts fully occluded objects in 3D, while SDXL and VLMs cannot.
- On the 2.5D task, DFM outperforms SDXL on most metrics, e.g., entropy 0.530 vs 0.617 on RealEstate10k.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SDXL outpainting instead of DFM for 2D detection | Unobserved object detection · 2D, RealEstate10k and NYU Depth V2 | nearest neighbor distance (%): 5.990 -> 0.446 (RealEstate10k); 6.217 -> 0.510 (NYU Depth V2) | Helped |
| Use DFM instead of SDXL outpainting | Unobserved object detection · 2.5D, RealEstate10k and NYU Depth V2 | nearest neighbor distance (%): 6.277 -> 5.128 (RealEstate10k); 5.245 -> 4.214 (NYU Depth V2) | Mixed |
| Reduce DFM camera poses from 4 to 3 (ablation) | DFM · Combined RealEstate10k and NYU Depth V2, 2D task | cross-entropy: 1.595 -> 1.944 | Hurt |
| Reduce DFM camera poses from 4 to 2 (ablation) | DFM · Combined RealEstate10k and NYU Depth V2, 2D task | false negative rate: 0.04 -> 0.88 | Hurt |
| Reduce DFM samples from 25 to 15, keeping 4 poses (ablation) | DFM · Combined RealEstate10k and NYU Depth V2, 2D task | cross-entropy: 1.595 -> 1.804 | Hurt |
| Reduce DFM samples from 25 to 10, keeping 4 poses (ablation) | DFM · Combined RealEstate10k and NYU Depth V2, 2D task | cross-entropy: 1.595 -> 2.342 | Hurt |
| Add target-object cues to SDXL prompts (ablation) | SDXL · Combined RealEstate10k and NYU Depth V2, 2D task | cross-entropy: 1.977 -> 1.244 | Helped |
| Remove prompts from SDXL (ablation) | SDXL · Combined RealEstate10k and NYU Depth V2, 2D task | cross-entropy: 1.244 -> 2.963 | Hurt |
| Remove target-object cues from SDXL prompts (ablation) | SDXL · Combined RealEstate10k and NYU Depth V2, 2.5D task | nearest neighbor distance (%): 5.884 -> 16.051 | Hurt |
Surprising findings
- SDXL outpainting performance depends heavily on object cues in the prompt; without prompts or with generic prompts, results degrade.
- DFM takes about 25 minutes per target pose per sample, roughly 48 days per dataset.
Limitations
- 3D diffusion inference is very slow and costly, limiting real-time robotics use.
- Outpainting methods require object prompts, which may bias the estimated distribution.
- VLM evaluation may be affected by memorization of test images.
Experimental setup
Datasets: RealEstate10k NYU Depth V2 MS-COCO
Benchmarks: RealEstate10k NYU Depth V2
Metrics: normalized cross-entropy normalized entropy normalized nearest neighbor distance false negative rate region-wise accuracy
Baselines: Uniform Oracle SDXL ChatGPT-4o Claude-3.5-Sonnet Gemini-1.5-Ultra LLaVa-v1.6-34b
Builds on: Diffusion with Forward Models SDXL YOLOv8 DepthAnything v2 COLMAP
Closest papers by meaning
- Generative Region-Language Pretraining for Open-Ended Object Detection (CVPR 2024 · similarity 0.88)
- Unsupervised Object Detection with Theoretical Guarantees (NeurIPS 2024 · similarity 0.87)
- Cubify Anything: Scaling Indoor 3D Object Detection (CVPR 2025 · similarity 0.86)
- Rooms from Motion: Un-posed Indoor 3D Object Detection as Localization and Mapping (NeurIPS 2025 · similarity 0.86)
- Uncertainty Meets Diversity: A Comprehensive Active Learning Framework for Indoor 3D Object Detection (CVPR 2025 · similarity 0.86)
- ImOV3D: Learning Open Vocabulary Point Clouds 3D Object Detection from Only 2D Images (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.