Research map / Detection and segmentation / CNN features

Not All Tokens Matter All The Time: Dynamic Token Aggregation Towards Efficient Detection Transformers

ICML 2025 · Detection and segmentation › CNN features · approach: Plain and high-resolution ViTs

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Dynamic DETR, a plug-in token aggregation method that sparsifies encoder tokens across FPN levels and encoder stages (proximal merging for low levels, holistic affinity merging for the top level) with a center-distance alignment loss, cutting DETR encoder FLOPs by roughly 40% at about 1 AP cost on COCO.

Problem

DETR encoders are a major computational bottleneck, and existing token sparsification methods for them use static, uniform retention ratios and ignore how token importance shifts across pyramid levels and encoder blocks, causing accuracy loss.

Approach

Stage-wise dynamic retention ratios derived from a per-stage importance ranking of pyramid levels; low-level tokens are merged within spatial windows (proximal aggregation) while the top level uses similarity-based holistic merging; a center-distance regularizer aligns pre- and post-sparsification representation centroids.

Key results

• On D-DETR, Dynamic DETR reduces FLOPs by 39.7% (179.0G to 107.9G) with AP 46.0 vs. 47.0 for the original model, matching Sparse DETR's AP at lower FLOPs.
• On DINO (COCO), Dynamic DINO reduces FLOPs by 42.0% (244.5G to 141.7G) with AP 50.2 vs. 50.9 original, outperforming Lite DETR and Focus DETR on FLOPs reduction.
• Ablation on DINO: the full method reaches 50.2 AP versus 46.6 AP for the variant without dynamic aggregation/multi-level design, and RCDR adds 0.6 AP.

Measured effects reported in the paper

Limitations

• Retention ratios and the importance ranking are fixed per stage rather than input-adaptive, as the authors note in future work.
• Accuracy drops of about 0.7 to 1.0 AP remain relative to the original models.
• Evaluation is limited to ResNet-50 and Swin-T backbones and COCO-family datasets, with PASCAL VOC and LVIS results shown only for DINO.

Experimental setup

Datasets: COCO 2017 LVIS v1.0 PASCAL VOC 2007

Benchmarks: COCO 2017 val LVIS v1.0 PASCAL VOC 2007

Metrics: AP AP50 AP75 APS APM APL AP_r AP_c AP_f mAP

Baselines: Sparse DETR Lite DETR Focus DETR DETR DAB-DETR DN-DETR Conditional DETR Anchor DETR

Builds on: Deformable DETR ToMe bipartite soft matching Lite DETR Sparse DETR DINO

Closest papers by meaning

• Sparse DETR: Efficient End-to-End Object Detection with Learnable Sparsity (ICLR 2022 · similarity 0.89)
• Token Cropr: Faster ViTs for Quite a Few Tasks (CVPR 2025 · similarity 0.86)
• Adaptive Length Image Tokenization via Recurrent Allocation (ICLR 2025 · similarity 0.86)
• ImagePiece: Content-aware Re-tokenization for Efficient Image Recognition (AAAI 2025 · similarity 0.86)
• Subobject-level Image Tokenization (ICML 2025 · similarity 0.85)
• CORE: Compact Object-centric REpresentations as a New Paradigm for Token Merging in LVLMs (CVPR 2026 · similarity 0.85)