Research map / 3D vision / 3D generation
BEHAVIOR Vision Suite: Customizable Dataset Generation via Simulation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces BEHAVIOR Vision Suite, a simulation-based toolkit (extended BEHAVIOR-1K assets plus a customizable generator on OmniGibson) for generating controlled synthetic vision datasets and evaluating vision models.
Problem
Real-world vision datasets offer fixed distributions and limited labels, making systematic, controlled evaluation of computer vision models hard; existing synthetic generators lack asset quality, physical plausibility, or customizability.
Approach
Combine an expanded, annotated 3D asset library (8,841 objects, 1,000 scene instances) with a simulator-based generator that exposes scene, object-state, and camera parameters, so researchers can vary one factor at a time and get dense labels plus physically plausible object states.
Key results
- Object detectors and segmenters show strong sensitivity to object visibility below 0.5 and to articulation (open states), with a notable negative correlation between articulation degree and detection AP.
- Sim2real relation classifier trained on 12.5k synthetic images reaches 0.863 precision, 0.817 recall, and 0.839 F1 on 910 real images, versus 0.293/0.282/0.271 for zero-shot CLIP.
- Relative model rankings on the synthetic scene-understanding data are reported to correlate with rankings on real datasets (COCO, NYUv2), used to argue the data is a faithful proxy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train the relation-prediction model on synthetic data rather than use zero-shot CLIP | Object relations prediction model · Real test set | F1: 0.271 -> 0.839 | Helped |
| Increase target-object articulation from closed toward open | Open-vocabulary detection models · BVS articulation axis | Average Precision | Hurt |
| Increase scene brightness from dark toward midpoint brightness 0.5 | Open-vocabulary detection models · BVS lighting axis | Average Precision | Helped |
| Reduce target-object visibility below 0.5 | Open-vocabulary detection models · BVS visibility axis | Average Precision | Hurt |
| Use extremely close or extremely distant camera zoom | Open-vocabulary detection models · BVS zoom axis | Average Precision | Hurt |
| Tilt the camera to look down at the target object | Open-vocabulary detection models · BVS pitch axis | Average Precision | Helped |
Surprising findings
- Detection performance is consistently better when the camera looks down at the target object, which the authors attribute to viewpoint bias in large real datasets.
- Model performance improves with brightness only up to a midpoint level (0.5) and then plateaus.
Limitations
- Sim2real evaluation of the relation classifier is limited to a small 910-image real test set and five relation labels.
- Realism is assessed only indirectly via model ranking correlation rather than a direct photorealism measure in the main text.
- The 1,000 scene instances are described as to-be-released, and the paper does not clearly state code or data availability.
Experimental setup
Datasets: BEHAVIOR-1K assets and scenes MS COCO NYUv2 Replica
Benchmarks: BVS parametric evaluation (articulation, lighting, visibility, zoom, pitch) BVS holistic scene understanding traversal videos BVS object state and relation prediction COCO ADE NYUv2 Replica
Metrics: Average Precision RMS AbsRel Log10 delta accuracy completion ratio accuracy precision recall F1
Baselines: GLIP RAM Grounding DINO ODISE OpenSeeD Grounding SAM DPT NVDS iDisc GradSLAM NICE-SLAM Zero-shot CLIP
Builds on: BEHAVIOR-1K OmniGibson CLIP 3DCC
Closest papers by meaning
- WorldSimBench: Towards Video Generation Models as World Simulators (ICML 2025 · similarity 0.86)
- Evaluating Multiview Object Consistency in Humans and Image Models (NeurIPS 2024 · similarity 0.86)
- VideoHallu: Evaluating and Mitigating Multi-modal Hallucinations on Synthetic Video Understanding (NeurIPS 2025 · similarity 0.85)
- Data-efficient Large Vision Models through Sequential Autoregression (ICML 2024 · similarity 0.85)
- Empowering World Models with Reflection for Embodied Video Prediction (ICML 2025 · similarity 0.85)
- WorldModelBench: Judging Video Generation Models As World Models (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.