Research map / Detection and segmentation / Open-set and few-shot detection
Visual Jenga: Discovering Object Dependencies via Counterfactual Inpainting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Visual Jenga, a task of progressively removing objects from a single image until only background remains, and a training-free counterfactual inpainting method that ranks removal order by inpainting diversity.
Problem
Scene understanding beyond object recognition: determining which objects can be removed from an image while keeping the scene physically and geometrically coherent, i.e., object dependency/support ordering.
Approach
Rank objects for removal by the asymmetry of inpainting diversity: an object whose masked region admits many plausible replacements (high diversity, low dependence on others) is removed first, using a generative inpainting model with CLIP and DINO similarity as a training-free proxy for conditional probabilities.
Key results
- On NYU-v2 pair-wise ordering, the method reaches 91.32% accuracy (610/668), versus 90.12% for small-to-large.
- On HardParse pair-wise ordering, accuracy is 65% (26/40 in the table; text says 28/40 at 70%), versus 52.5% for the best heuristic baseline.
- Full scene decomposition succeeds on 71.43% (40/56) of scenes, versus 42.85% for the best heuristic (small-to-large).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add DINO similarity scores (ablation) | Visual Jenga · NYU-v2 | accuracy: 89.52% -> 91.32% | Helped |
| add DINO similarity scores (ablation) | Visual Jenga · HardParse | accuracy: 55% -> 65% | Helped |
| add CLIP similarity scores (ablation) | Visual Jenga · NYU-v2 | accuracy: 90.27% -> 91.32% | Helped |
| add CLIP similarity scores (ablation) | Visual Jenga · HardParse | accuracy: 57.5% -> 65% | Helped |
| increase inpainting samples from N=2 to N=4 (ablation) | Visual Jenga · HardParse | accuracy: 50% -> 50% | No clear effect |
| increase inpainting samples from N=4 to N=8 (ablation) | Visual Jenga · HardParse | accuracy: 50% -> 62.5% | Helped |
| increase inpainting samples from N=8 to N=16 (ablation) | Visual Jenga · HardParse | accuracy: 62.5% -> 65% | Helped |
| use counterfactual inpainting ordering instead of Small-to-Large | Visual Jenga · NYU-v2 | accuracy: 90.12% -> 91.32% | Helped |
| use counterfactual inpainting ordering instead of Front-to-Back | Visual Jenga · HardParse | accuracy: 52.5% -> 65% | Helped |
| use counterfactual inpainting ordering instead of Small-to-Large | Visual Jenga · Full Scene Decomposition | success rate: 42.85% -> 71.43% | Helped |
Surprising findings
- Simple heuristics such as small-to-large reach 90.12% on NYU-v2 pair-wise ordering, nearly matching the proposed method, but collapse on harder scenes.
- Using CLIP or DINO alone yields lower HardParse accuracy than combining both, and DINO tends to favor smaller objects.
Limitations
- Slow, requiring multiple inpainting passes per object, and produces only a single sequential removal order.
- Relies on Molmo and SAM for object detection and segmentation; fails when they fail (e.g., missed glasses, partial monitor masks).
- Purely statistical co-occurrence signal without explicit physical reasoning; VLM baselines produced largely unusable visual outputs.
Experimental setup
Datasets: NYU Depth V2 HardParse (40 internet images, curated by authors) Full scene decomposition set (56 scenes, own photos and internet)
Benchmarks: NYU-v2 pair-wise object ordering HardParse pair-wise object ordering Full scene decomposition (56 scenes)
Metrics: pair-wise accuracy pass rate of full scene decomposition (human pass/fail)
Baselines: Top-to-Bottom heuristic Small-to-Large heuristic Front-to-Back heuristic ChatGPT + DALLE ChatGPT + InstructPix2Pix ChatGPT + Molmo + SAM + Adobe Firefly
Builds on: Reichenbach's principle of the common cause Biederman scene perception constraints Visual Genome
Closest papers by meaning
- InstaInpaint: Instant 3D-Scene Inpainting with Masked Large Reconstruction Model (NeurIPS 2025 · similarity 0.85)
- SceneFun3D: Fine-Grained Functionality and Affordance Understanding in 3D Scenes (CVPR 2024 · similarity 0.83)
- Common Inpainted Objects In-N-Out of Context (CVPR 2026 · similarity 0.83)
- MVInpainter: Learning Multi-View Consistent Inpainting to Bridge 2D and 3D Editing (NeurIPS 2024 · similarity 0.83)
- Instruct2See: Learning to Remove Any Obstructions Across Distributions (ICML 2025 · similarity 0.83)
- $\textit{Bifr\"ost}$: 3D-Aware Image Compositing with Language Instructions (NeurIPS 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.