Research map / Detection and segmentation / CNN features
Joint detection and instance segmentation
Couple object detection and instance segmentation in unified or query-based models.
39 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in CNN features, keyword cluster “instance · panoptic · mask”
State History Benchmarks Tried Checklist Preprints Export
Where it stands
Quiet None of the tracked benchmarks has a report from 2025–26.
7 papers in 2023–24 and 3 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 2.1% to 0.9%.
No longer reported by papers here: COCO test-dev (AP, last in 2022).
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · Multi-Scale Patch Aggregation (MPA) for Simultaneous Detection and Segmentation (CVPR 2016 · 89 citations)
- 2017 · Associative Embedding: End-to-End Learning for Joint Detection and Grouping (NeurIPS 2017 · 554 citations)
- 2018 · Learning Deep Structured Active Contours End-to-End (CVPR 2018 · 155 citations)
- 2019 · Devil in the Details: Towards Accurate Single and Multiple Human Parsing (AAAI 2019 · 248 citations)
- 2020 · D2Det: Towards High Quality Object Detection and Instance Segmentation (CVPR 2020 · 213 citations)
- 2021 · DetectoRS: Detecting Objects With Recursive Feature Pyramid and Switchable Atrous Convolution (CVPR 2021 · 1,093 citations)
- 2022 · InstaFormer: Instance-Aware Image-to-Image Translation With Transformer (CVPR 2022 · 55 citations)
- 2023 · Learning Dynamic Query Combinations for Transformer-based Object Detection and Segmentation (ICML 2023 · 2 citations)
- 2024 · MGMap: Mask-Guided Learning for Online Vectorized HD Map Construction (CVPR 2024 · 47 citations)
- 2025 · RMP-SAM: Towards Real-Time Multi-Purpose Segment Anything (ICLR 2025 · 4 citations)
- 2026 · Differentiable Laplacian Matrix Guided Superpixel Segmentation (CVPR 2026)
Earliest papers here
- Multi-Scale Patch Aggregation (MPA) for Simultaneous Detection and Segmentation (CVPR 2016 · 89 citations)
- Associative Embedding: End-to-End Learning for Joint Detection and Grouping (NeurIPS 2017 · 554 citations)
- MIML-FCN+: Multi-Instance Multi-Label Learning via Fully Convolutional Networks With Privileged Information (CVPR 2017 · 72 citations)
Benchmark records
The best number each year that beat every earlier one, among papers of this approach reporting their own method. Numbers are extracted by AI from each paper's text; check the paper before citing.
COCO test-dev · AP (higher is better)
4 papers, 2018–2022 · not reported since 2022
| Year | Best so far | Set by |
|---|---|---|
| 2018 | 46.1 | SNIPER: Efficient Multi-Scale Training (NeurIPS 2018 · 55 citations) |
| 2020 | 52.1 | RepPoints v2: Verification Meets Regression for Object Detection (NeurIPS 2020 · 81 citations) |
| 2022 | 72.3 | QueryPose: Sparse Multi-Person Pose Regression via Spatial-Aware Part-Level Query (NeurIPS 2022 · 9 citations) |
What has been tried
37 papers here report 419 measured changes (a component added, removed or swapped, compared with a baseline): helped 79%, no clear effect 4%, hurt 15%, mixed 2%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Add instance-level MAI | nuScenes val, camera, 60m × 30m · mAP | 57.6 -> 59.5 | Helped | MGMap: Mask-Guided Learning for Online Vectorized HD Map Construction (CVPR 2024 · 47 citations) |
| add two-branch decoding to the combined instance- and boundary-level framework | MultiTHUMOS · DetmAP average | 27.04 -> 29.04 | Helped | Dual DETRs for Multi-Label Temporal Action Detection (CVPR 2024 · 32 citations) |
| Add semantic removal to the shadow representation | Visual Genome (VG) · mIoU | 22.7 -> 25.4 | Helped | Shadow-Enlightened Image Outpainting (CVPR 2024 · 8 citations) |
| increase Transformer decoder depth from 8 to 10 layers | 50Salads · accuracy | 88.4 -> 89.5 | Helped | Efficient Temporal Action Segmentation via Boundary-aware Query Voting (NeurIPS 2024 · 6 citations) |
| add DI and BATO | BDD100K, 68 epochs · APbox | 27.8 -> 29.5 | Helped | DI-MaskDINO: A Joint Object Detection and Instance Segmentation Model (NeurIPS 2024 · 5 citations) |
| Add video training to image-only training | COCO-Panoptic · PQ | 36.6 -> 36.2 | Hurt | RMP-SAM: Towards Real-Time Multi-Purpose Segment Anything (ICLR 2025 · 4 citations) |
| Replace ROI-Align and ROI-Unpool with masked attention | ROICtrl-Bench AVG · mIoU | 0.652 -> 0.628 | Hurt | ROICtrl: Boosting Instance Control for Visual Generation (CVPR 2025 · 1 citation) |
What reviewers will expect
Datasets and baselines named in the 7 papers of this approach from 2024–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Baselines
- Mask2Former · 2 of 7 papers
Recent preprints
Loading recent arXiv preprints…
Use it
Download BibTeX of 28 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Boundary-aware instance segmentation (83 papers)
- Unified panoptic segmentation (33 papers)
Elsewhere in the topic
- Context modeling for semantic segmentation (241 papers)
- Region-based and query-based detectors (183 papers)
- Local features and context in CNNs (170 papers)
- Context for hard detection conditions (133 papers)
- Domain-specific segmentation and detection (118 papers)
- Convolution-transformer backbones (101 papers)
- Box regression and loss design (94 papers)
- Efficient DETR query design (85 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).