Research map / Detection and segmentation / Open-set and few-shot detection
Holistic Order Prediction in Natural Scenes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes InstaFormer, a Mask2Former-based network that predicts full occlusion and depth order matrices for all instances in an RGB image in a single forward pass, benchmarked on InstaOrder.
Problem
Instance-wise occlusion and depth ordering prediction is usually done pairwise, requiring ground-truth instance masks as input and a quadratic number of forward passes over instance pairs.
Approach
Reformulates pairwise edge-level order prediction as a single adjacency-matrix prediction: object queries and max-pooled latent mask descriptors interact through transformer layers, and their dot product yields the full order matrix from an RGB image alone.
Key results
- InstaFormer(o) reaches 81.89 occlusion F1 on InstaOrder, slightly above InstaOrderNet(o,d) at 81.86 and the best overall.
- InstaFormer(o,d) achieves 7.90 distinct, 24.68 overlap and 13.30 all WHDR for depth, the best depth results in the table (InstaOrderNet(o,d) 15.99 all).
- Pairwise baselines need ground-truth masks as input and quadratic forward passes; InstaFormer runs in a single forward pass with runtime constant in number of instances.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| predict occlusion order holistically from RGB with InstaFormerᵒ rather than using InstaOrderNetᵒ,d | InstaOrder · InstaOrder | F1: 81.86 -> 81.89 | Helped |
| predict depth order holistically with InstaFormerᵈ rather than InstaOrderNetᵈ | InstaFormerᵈ · InstaOrder | WHDR all: 17.51 -> 13.73 | Helped |
| jointly predict occlusion and depth order rather than train the occlusion-specific model | InstaFormer · InstaOrder | F1: 81.89 -> 81.37 | Hurt |
| jointly predict depth with occlusion rather than train the depth-specific model | InstaFormer · InstaOrder | WHDR all: 13.73 -> 13.30 | Helped |
| use Swin-L†200 instead of Swin-B†100 (ablation) | InstaFormerᵒ,d · InstaOrder | F1: 80.34 -> 81.37 | Helped |
| use Swin-L†200 instead of Swin-B†100 (ablation) | InstaFormerᵒ,d · InstaOrder | WHDR all: 14.06 -> 13.30 | Helped |
| combine object queries with latent mask descriptors instead of using queries for both token types (ablation) | InstaFormerᵒ,d Swin-T · InstaOrder | F1: 77.92 -> 79.74 | Helped |
| combine object queries with latent mask descriptors instead of using queries for both token types (ablation) | InstaFormerᵒ,d Swin-T · InstaOrder | WHDR all: 23.21 -> 14.03 | Helped |
| add one transformer layer before max pooling (ablation) | InstaFormerᵒ,d Swin-T · InstaOrder | WHDR all: 21.88 -> 14.03 | Helped |
| use 2 transformer layers instead of 1 before max pooling (ablation) | InstaFormerᵒ,d Swin-T · InstaOrder | WHDR all: 14.03 -> 21.48 | Hurt |
| add adapters to the segmentation backbone (ablation) | InstaFormerᵒ · InstaOrder | F1: 79.71 -> 81.89 | Helped |
| add adapters to the segmentation backbone (ablation) | InstaFormerᵈ · InstaOrder | WHDR all: 23.52 -> 13.73 | Helped |
Surprising findings
- Zero-shot LLaVA performs worst among benchmarked methods, and fine-tuning on 2% of the VQA data substantially improves it but it remains behind visual experts.
- Depth order prediction is little affected by segmentation backbone size, plateauing around 13.7 WHDR, suggesting depth needs no fine-grained edge information.
Limitations
- InstaOrder annotations are limited to 10 instances per image, limiting evaluation generality.
- Failure cases include instance mixup and segmentation failures producing duplicate instances.
- Occlusion F1 is only marginally better than pairwise baselines, and joint training is harder to optimize for occlusion.
Experimental setup
Datasets: InstaOrder COCO
Benchmarks: InstaOrder InstaOrder-Vqa
Metrics: precision recall F1-score WHDR
Baselines: LLaVA 1.5 (zero-shot) LLaVA 1.5 (fine-tuned) Area heuristic Y-axis heuristic PCNet-M OrderNet InstaOrderNet MiDaS Foundation pipeline (Grounded SAM + Depth Anything V2) Mask2Order
Builds on: Mask2Former DETR InstaOrderNet LLaVA
Closest papers by meaning
- SyFormer: Structure-Guided Synergism Transformer for Large-Portion Image Inpainting (AAAI 2024 · similarity 0.84)
- REOrdering Patches Improves Vision Models (NeurIPS 2025 · similarity 0.83)
- InstaGen: Enhancing Object Detection by Training on Synthetic Dataset (CVPR 2024 · similarity 0.83)
- Order-aware Interactive Segmentation (ICLR 2025 · similarity 0.83)
- Vision Transformers Learn Gestalt-Like Figure-Ground Cues from Natural Images (NeurIPS 2026 · similarity 0.83)
- Inst3D-LMM: Instance-Aware 3D Scene Understanding with Multi-modal Instruction Tuning (CVPR 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.