Research map / Detection and segmentation / Open-set and few-shot detection
Seeing the Unseen: Visual Common Sense for Semantic Placement
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Semantic Placement (predicting masks where an absent object could be placed), builds a ~1.3M-image inpainting-based dataset from LAION and trains a CLIP-UNet model, pretrained on LAION-SP and finetuned on HSSD synthetic scenes.
Problem
Predicting semantically plausible placement regions in an image for an object that is not visible in the image, which existing VLMs struggle to localize precisely in pixel space.
Approach
Generate paired with/without-object training data automatically by detecting real objects in web images and inpainting them out (with artifact-mitigating diffusion augmentation and filtering), then train a frozen-CLIP ResNet50 with a language-conditioned UNet decoder, followed by finetuning on programmatically rendered HSSD scenes.
Key results
- CLIP-UNet (LAION-SP→HSSD) is preferred in 43.7% of human rankings on real LAION images versus 9.4% for GPT-4V and 21.5% for the LLM+Detector baseline.
- On HSSD val, CLIP-UNet reaches 18.5% target precision, versus 16.2% for the HSSD-only model and 10.1% for the LAION-SP-only model.
- In embodied semantic placement on HSSD, the CLIP-UNet policy achieves 12.5% success, versus 10.5% for LLM+Detector and 9.0% for LLaVA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use CLIP-UNet pretrained on LAION-SP and finetuned on HSSD instead of LLM + Detector | Semantic Placement · LAION-SP Val | HP: 21.5 -> 43.7 | Helped |
| use CLIP-UNet pretrained on LAION-SP and finetuned on HSSD instead of LLM + Detector | Semantic Placement · HSSD Val | HP: 29.8 -> 31.3 | Helped |
| use CLIP-UNet pretrained on LAION-SP and finetuned on HSSD instead of LLM + Detector | Semantic Placement · HSSD Val | TrP: 10.1 -> 18.5 | Helped |
| use CLIP-UNet pretrained on LAION-SP and finetuned on HSSD instead of LLM + Detector | Semantic Placement · HSSD Val | RSP: 41.0 -> 24.9 | Hurt |
| use CLIP-UNet pretrained on LAION-SP and finetuned on HSSD instead of LLM + Detector | Semantic Placement · HSSD Val | RSR: 38.2 -> 35.3 | Hurt |
| use CLIP-UNet pretrained on LAION-SP and finetuned on HSSD instead of LLaVA | Semantic Placement · LAION-SP Val | HP: 4.9 -> 43.7 | Helped |
| use CLIP-UNet pretrained on LAION-SP and finetuned on HSSD instead of LLaVA | Semantic Placement · HSSD Val | TrP: 0.0 -> 18.5 | Helped |
| pretrain CLIP-UNet on LAION-SP before HSSD training (ablation) | CLIP-UNet · HSSD VAL | TrP: 16.2 -> 18.5 | Helped |
| pretrain CLIP-UNet on LAION-SP before HSSD training (ablation) | CLIP-UNet · HSSD VAL | RSP: 26.6 -> 24.9 | Hurt |
| pretrain CLIP-UNet on LAION-SP before HSSD training (ablation) | CLIP-UNet · HSSD VAL | RSR: 36.5 -> 35.3 | Hurt |
| finetune LAION-SP-pretrained CLIP-UNet on HSSD (ablation) | CLIP-UNet · HSSD VAL | TrP: 10.1 -> 18.5 | Helped |
| finetune LAION-SP-pretrained CLIP-UNet on HSSD (ablation) | CLIP-UNet · HSSD VAL | RSP: 23.7 -> 24.9 | Helped |
Surprising findings
- Models trained on inpainted images without quality enhancement yield near-zero true positives zero-shot on HSSD, indicating latching onto inpainting artifacts.
- Off-the-shelf VLMs such as GPT-4V give meaningful language responses but fail to localize placements precisely in pixel space.
Limitations
- Embodied placement success is low (12.5%), with most failures attributed to control policy and fine-grained navigation.
- Precision on all visible receptacles is lower than the LLM+Detector baseline, suggesting false positives near receptacles not grounded to surfaces.
- Training relies on inpainting models whose artifacts the model can latch onto.
Experimental setup
Datasets: LAION HSSD
Benchmarks: LAION-SP Val HSSD Val
Metrics: human preference target precision receptacle surface precision receptacle surface recall success rate
Baselines: LLM + Detector LLaVA-1.5 GPT-4V Ours (HSSD)
Builds on: CLIP CLIPort LingUNet Detic SAM
Closest papers by meaning
- Mind the GAP: Glimpse-based Active Perception improves generalization and sample efficiency of visual reasoning (ICLR 2025 · similarity 0.86)
- NL-Eye: Abductive NLI For Images (ICLR 2025 · similarity 0.85)
- RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Robotics (CVPR 2025 · similarity 0.85)
- CoSpace: Benchmarking Continuous Space Perception Ability for Vision-Language Models (CVPR 2025 · similarity 0.85)
- VisRes Bench: On Evaluating the Visual Reasoning Capabilities of VLMs (CVPR 2026 · similarity 0.85)
- FirePlace: Geometric Refinements of LLM Common Sense Reasoning for 3D Object Placement (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.