Research map / Detection and segmentation / CNN features
DI-MaskDINO: A Joint Object Detection and Instance Segmentation Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DI-MaskDINO, which adds a De-Imbalance module (residual double-selection over feature tokens) and a Balance-Aware Tokens Optimization module to MaskDINO to reduce the detection-segmentation performance imbalance in the first decoder layer, improving joint detection and instance segmentation on COCO and BDD100K.
Problem
Joint object detection and instance segmentation suffers from a detection-segmentation performance imbalance at the first transformer decoder layer, which is hypothesized to cap final performance of models like MaskDINO.
Approach
Replace MaskDINO's direct top-k selection of decoder queries with a residual double-selection mechanism (token self-attention interaction then a second top-k selection, fused with the initial tokens via cross-attention) to produce a balance-aware query, and use that query to guide optimization of the encoder tokens fed to the decoder as keys and values.
Key results
- On COCO with ResNet50 and 12 epochs, DI-MaskDINO reaches 46.9 AP box and 42.3 AP mask, +1.2 and +0.9 over MaskDINO.
- On BDD100K with ResNet50 and 68 epochs, DI-MaskDINO reaches 29.5 AP box and 25.7 AP mask versus MaskDINO's 28.1 and 25.3.
- Under SwinL backbone with 50 epochs on COCO, DI-MaskDINO achieves 57.8 AP box and 51.8 AP mask versus MaskDINO's 56.8 and 51.0.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add DI-MaskDINO modules to MaskDINO | MaskDINO · COCO val, ResNet50, 12 epochs | APbox: 45.7 -> 46.9 | Helped |
| add DI-MaskDINO modules to MaskDINO | MaskDINO · COCO val, ResNet50, 12 epochs | APmask: 41.4 -> 42.3 | Helped |
| add DI-MaskDINO modules to MaskDINO | MaskDINO · BDD100K val, ResNet50, 68 epochs | APbox: 28.1 -> 29.5 | Helped |
| add DI-MaskDINO modules to MaskDINO | MaskDINO · BDD100K val, SwinL, 68 epochs | APmask: 27.0 -> 27.9 | Helped |
| add DI and BATO (ablation) | DI-MaskDINO without DI and BATO · BDD100K, 68 epochs | APbox: 27.8 -> 29.5 | Helped |
| add DI and BATO (ablation) | DI-MaskDINO without DI and BATO · COCO, 12 epochs | APbox: 45.6 -> 46.9 | Helped |
| add BATO (ablation) | DI-MaskDINO without BATO · BDD100K, 68 epochs | APbox: 28.8 -> 29.5 | Helped |
| add DI (ablation) | DI-MaskDINO without DI · BDD100K, 68 epochs | APbox: 28.3 -> 29.5 | Helped |
| use residual balance-aware query Q_bal rather than second-selected tokens as BATO guidance (ablation) | DI-MaskDINO · BDD100K | APbox: 28.9 -> 29.5 | Helped |
| add guiding token generation (ablation) | BATO · BDD100K | APbox: 28.6 -> 29.5 | Helped |
| use triple-selection instead of double-selection (ablation) | DI-MaskDINO · BDD100K | APbox: 29.5 -> 29.3 | Hurt |
| increase decoder layers from 6 to 9 (ablation) | DI-MaskDINO · BDD100K | APbox: 29.5 -> 28.9 | Hurt |
Surprising findings
- Standard MaskDINO's beginning-layer detection lags segmentation, and the paper claims this imbalance caps final performance (upper bound), a claim supported by imbalance-tolerance tests where MaskDINO's box AP drops 21.8% under a position-token constraint.
- DI-MaskDINO with 3 decoder layers (45.8 AP box, 41.3 AP mask on COCO) is comparable to MaskDINO with 9 layers (45.7, 41.4).
Limitations
- Applicable only to joint object detection and instance segmentation; not evaluated for semantic or panoptic segmentation.
- Swin-L experiments were run on A6000 GPUs with batch size 4, so the reproduced MaskDINO numbers are lower than originally reported.
- Gains on COCO at 50 epochs are small for AP box (+0.2 with ResNet50).
Experimental setup
Datasets: COCO train2017 BDD100K
Benchmarks: COCO val2017 BDD100K validation set
Metrics: AP box AP mask AP small/medium/large FPS
Baselines: MaskDINO DINO Mask2Former Mask RCNN HTC SOLQ
Builds on: MaskDINO DINO Deformable DETR
Closest papers by meaning
- Mask DINO: Towards a Unified Transformer-Based Framework for Object Detection and Segmentation (CVPR 2023 · similarity 0.88)
- The Devil is in the Object Boundary: Towards Annotation-free Instance Segmentation using Foundation Models (ICLR 2024 · similarity 0.85)
- EfficientSAM: Leveraged Masked Image Pretraining for Efficient Segment Anything (CVPR 2024 · similarity 0.85)
- MaskDiff: Modeling Mask Distribution with Diffusion Probabilistic Model for Few-Shot Instance Segmentation (AAAI 2024 · similarity 0.85)
- Segment Anything, Even Occluded (CVPR 2025 · similarity 0.85)
- Triple Feature Disentanglement for One-Stage Adaptive Object Detection (AAAI 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.