Research map / Detection and segmentation / CNN features
DECO: Unleashing the Potential of ConvNets for Query-based Detection and Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds a DETR-style query-based detector (DECO) and a SAM mask decoder using only convolutions, via a novel InterConv mechanism that replaces self- and cross-attention with depthwise/1x1 convs plus upsampling and pooling.
Problem
Query-based (DETR-style) detection and segmentation normally require transformer decoders with attention, which are hard to support on some AI chips; the paper asks whether pure ConvNets can replace attention-based decoders while keeping DETR's query-based, NMS-free properties.
Approach
InterConv reshapes object queries into a 2D grid, upsamples them to the encoder feature size, fuses them with image features, applies large-kernel depthwise convolution, and max-pools back, emulating cross-attention with convolutions; Self-InterConv uses depthwise and 1x1 convs on the reshaped queries to emulate self-attention.
Key results
- DECO+ with ResNet-18 achieves 40.5 AP at 66 FPS and with ResNet-50 achieves 47.8 AP at 34 FPS on COCO val (V100).
- DECO+ with ConvNeXt-B reaches 50.6 AP at 20 FPS, 0.2 AP above Focus-DETR at the same FPS.
- DECO-TinySAM gets 41.4 zero-shot COCO AP vs 41.9 for TinySAM, with 29 ms vs 34 ms mobile latency.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace DETR's transformer encoder and decoder with the DECO convolutional encoder and decoder | DETR with ResNet-50 · COCO 2017 | FPS: 28 -> 35 | Helped |
| Replace TinySAM's mask decoder with the DECO InterConv decoder | TinySAM · COCO box-prompted zero-shot instance segmentation | Mobile latency: 34 -> 29 | Helped |
| Replace TinySAM's mask decoder with the DECO InterConv decoder | TinySAM · COCO box-prompted zero-shot instance segmentation | AP: 41.9 -> 41.4 | Hurt |
| Use dynamic upsampling to match the encoder embedding size instead of a fixed-size resize (ablation) | DECO CIM · COCO | AP: 0.5 AP drop | Helped |
| Increase the number of decoder layers (ablation) | DECO · COCO | AP | Helped |
| Use a 5×5 decoder kernel instead of the selected 9×9 kernel (ablation) | DECO · COCO | AP: 37.8 vs 38.6 | Hurt |
| Increase the decoder kernel size beyond 9×9 (ablation) | DECO · COCO | AP: 38.6; 38.4; 38.6 | No clear effect |
| Use element-wise multiplication rather than addition for CIM fusion (ablation) | DECO CIM · COCO | AP | Hurt |
| Use addition rather than concatenation followed by convolution for CIM fusion (ablation) | DECO CIM · COCO | AP | No clear effect |
| Use a 10×10 query shape rather than alternative shapes (ablation) | DECO with 100 queries · COCO | AP | Helped |
| Use query shape 30×10 rather than 20×15 (ablation) | DECO with 300 queries · COCO | AP | Helped |
| Replace Self-InterConv depthwise convolution with DCNv3 (ablation) | DECO+ with ResNet-18 · COCO 2017 | AP: +1.0 AP | Helped |
Surprising findings
- Vanilla DECO with ResNet-50 has higher GFLOPs than DETR yet runs faster on V100 (35 vs 28 FPS), suggesting convolutions are more GPU-friendly than attention.
- Element-wise multiplication fusion did not outperform add fusion in CIM.
Limitations
- Vanilla DECO without multi-scale fusion trails DETR-variant accuracy; gains rely on RT-DETR-style multi-scale fusion (DECO+).
- Segment anything evaluation is a proof of concept without careful tuning, and is slightly below TinySAM in AP.
- Deformable attention and denoising training not adapted to DECO, leaving them for future work.
Experimental setup
Datasets: COCO 2017
Benchmarks: COCO 2017 val
Metrics: AP AP50 AP75 FPS GFLOPs mobile latency
Baselines: DETR Deformable-DETR RT-DETR YOLOS Sparse R-CNN OneNet DeFCN Focus-DETR TinySAM Faster R-CNN
Builds on: DETR ConvNeXt RT-DETR SAM
Closest papers by meaning
- A ConvNet for the 2020s (CVPR 2022 · similarity 0.89)
- RelationNet++: Bridging Visual Representations for Object Detection via Transformer Decoder (NeurIPS 2020 · similarity 0.87)
- ConvNet vs Transformer, Supervised vs CLIP: Beyond ImageNet Accuracy (ICML 2024 · similarity 0.86)
- BEVNeXt: Reviving Dense BEV Frameworks for 3D Object Detection (CVPR 2024 · similarity 0.86)
- MI-DETR: An Object Detection Model with Multi-time Inquiries Mechanism (CVPR 2025 · similarity 0.85)
- IFORMER: INTEGRATING CONVNET AND TRANSFORMER FOR MOBILE APPLICATION (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.