Research map / Detection and segmentation / CNN features
Union-over-Intersections: Object Detection beyond Winner-Takes-All
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Union-over-Intersections (UoI), which regresses each proposal only to its intersection with the ground truth and merges grouped proposals' intersections by union, as a plug-and-play change to detectors.
Problem
Box regression in object detectors forces each proposal or query to align with the entire ground truth box, which is hard when proposals cover only part of the object, and winner-takes-all NMS discards information from other proposals in the same region.
Approach
Replace full-box regression with intersection-of-proposal-and-ground-truth regression, then group proposals as in NMS and take the union (min/max) of their regressed intersections as the final box, followed by a refinement regression.
Key results
- Faster R-CNN R-101-fpn improves from 39.4 to 40.3 mAP on COCO with UoI; Cascade R-CNN R-101 improves from 42.5 to 43.1 mAP.
- Deformable DETR R-50 improves from 44.3 to 44.8 mAP on COCO.
- On Faster R-CNN, localization mIoU rises from 53.7% to 64.4% while classification accuracy stays near 76.4% to 76.5%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply intersection-based regression and union-over-intersections instead of traditional box selection | Faster R-CNN, Mask R-CNN, Cascade R-CNN, YOLOv3, and Deformable DETR · MS-COCO 2017 detection | mAP: 37.4 -> 38.1; 39.4 -> 40.3; 38.2 -> 38.8; 39.8 -> 40.9; 42.5 -> 43.1; 33.7 -> 34.5; 44.3 -> 44.8 | Helped |
| Apply UoI to Mask R-CNN | Mask R-CNN with R-50 and R-101 backbones · MS-COCO 2017 instance segmentation | mAP: 34.7 -> 35.3; 36.0 -> 36.8 | Helped |
| Apply UoI | Faster R-CNN · MS-COCO, classification and localization analysis | Localization mIoU [%]: 53.7 -> 64.4 | Helped |
| Replace traditional box proposal selection with UoI | Faster R-CNN · MS-COCO | LRP (Error): 67.6 -> 65.3 | Helped |
| Use intersection-based regression and intersection grouping rather than full-object regression (ablation) | Faster R-CNN · MS-COCO | mAP: 35.8 -> 38.1 | Helped |
| Add a second regression stage to baseline without UoI (ablation) | Faster R-CNN · MS-COCO | mAP: 37.4 -> 37.4 | No clear effect |
| Use UoI's second regression stage to refine combined intersections (ablation) | Faster R-CNN · MS-COCO | mAP: 37.4 -> 38.1 | Helped |
| Add a classification head for combined boxes (ablation) | Faster R-CNN w/ UoI · MS-COCO | mAP: 38.1 -> 38.2 | Helped |
| Use five proposals per group rather than fewer or more (ablation) | UoI · MS-COCO detection | mAP | Mixed |
| Lower the grouping IoU threshold to 0.3 rather than use 0.5 (ablation) | UoI with Faster R-CNN R-101 · MS-COCO | mAP: 40.3 -> 39.5 | Hurt |
| Raise the grouping IoU threshold to 0.7 rather than use 0.5 (ablation) | UoI with Faster R-CNN R-101 · MS-COCO | mAP: 40.3 -> 39.9 | Hurt |
| Combine UoI with test-time augmentation instead of no augmentation (ablation) | Faster R-CNN · MS-COCO | mAP: 38.1 -> 38.2; 38.1 -> 38.6; 38.1 -> 39.3 | Helped |
Surprising findings
- Adding a plain second regression head to Faster R-CNN gives no gain (37.4 to 37.4 mAP), while UoI's second stage does help.
- Full-object regression combined with grouping drops Faster R-CNN to 35.8 mAP, below the baseline of 37.4.
Limitations
- For small objects with only one proposal available, the method defaults to the baseline.
- Crowded same-class objects can merge into a single box.
- Gains depend on having multiple proposals per object; grouping of nearby instances is imperfect.
Experimental setup
Datasets: MS-COCO 2017 PASCAL VOC 2007 PASCAL VOC 2012
Benchmarks: MS-COCO 2017 PASCAL VOC
Metrics: mAP AP50 AP75 AP_S AP_M AP_L mIoU LRP
Baselines: Faster R-CNN Mask R-CNN Cascade R-CNN YOLOv3 Deformable DETR Soft-NMS Cluster-NMS box voting one-to-few label assignment
Builds on: Faster R-CNN Mask R-CNN Cascade R-CNN YOLOv3 Deformable DETR
Closest papers by meaning
- UNION: Unsupervised 3D Object Detection using Object Appearance-based Pseudo-Classes (NeurIPS 2024 · similarity 0.86)
- Pixel-level Quality Assessment for Oriented Object Detection (AAAI 2026 · similarity 0.86)
- Recognize Any Regions (NeurIPS 2024 · similarity 0.85)
- Generalized Intersection Over Union: A Metric and a Loss for Bounding Box Regression (CVPR 2019 · similarity 0.85)
- LocNet: Improving Localization Accuracy for Object Detection (CVPR 2016 · similarity 0.85)
- DiffuBox: Refining 3D Object Detection with Point Diffusion (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.