Research map / Image generation / Autoregressive and latent generation
Efficient Deformable ConvNets: Rethinking Dynamic and Sparse Operator for Vision Applications
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DCNv4, a faster deformable convolution operator that removes softmax normalization of aggregation weights and optimizes GPU memory access, and evaluates it across classification, detection, segmentation, 3D detection, and diffusion.
Problem
Deformable convolution (DCNv3) is slow due to memory-access overhead and converges more slowly than attention and convolution, limiting its adoption as a backbone operator.
Approach
Remove softmax normalization from DCNv3's spatial aggregation weights so they are unbounded and dynamic like convolution weights, and restructure the CUDA kernel so one thread processes all channels in a group, sharing offsets and weights, with vectorized loads and half precision.
Key results
- DCNv4 operator achieves more than 3x forward speedup over DCNv3 (e.g., 0.606 ms vs 1.45 ms at 56x56x128).
- FlashInternImage-T reaches 83.6% ImageNet top-1 with 2316/3154 img/s throughput vs InternImage-T 83.5% at 1409/1746, i.e. 64-80% faster.
- Latent diffusion U-Net with DCNv4 improves FID from 2.94 to 2.44 with fewer parameters (566M vs 860M).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Remove softmax normalization from spatial aggregation weights (ablation) | DCNv3 · ImageNet-1K classification | convergence speed and final performance | Helped |
| Apply softmax to the depthwise convolution weights (ablation) | ConvNeXt · ImageNet-1K, 300 epochs | accuracy: 83.8 -> 81.6 | Hurt |
| Replace DCNv3 with DCNv4 | InternImage · ImageNet-1K classification | throughput: +64%/ + 80% | Helped |
| Replace DCNv3 with DCNv4 | InternImage · ImageNet-1K classification | Top-1 accuracy: 83.5 -> 83.6; 84.2 -> 84.4; 84.9 -> 84.9; 87.7 -> 88.1 | Mixed |
| Replace depthwise convolution with DCNv4 (ablation) | ConvNeXt-B · ImageNet-1K | Top-1 accuracy: 83.8 -> 84.0 | Helped |
| Replace attention module with DCNv4 (ablation) | ViT-B · ImageNet-1K | Top-1 accuracy: 81.8 -> 81.9 | Helped |
| Replace attention and regular 3 × 3 convolution with DCNv4 (ablation) | U-Net · ImageNet 256x256 class-conditional generation | FID: 2.94 -> 2.44 | Helped |
| Replace DCNv3 with DCNv4 | DCN operator · 56 × 56 × 128 input, FP16 | runtime: 1.52 -> 0.404 | Helped |
| Remove the softmax operation and improve micro design by merging linear layers and removing LN-GELU (ablation) | DCNv3 module · DCN runtime, 56 × 56 × 128 | module runtime (ms): 3.28 -> 2.12 | Helped |
| Process 8 channels per thread to reduce redundant memory access (ablation) | DCNv3 kernel · DCN runtime, 56 × 56 × 128 | module runtime (ms): 2.12 -> 2.20 | Hurt |
| Reuse bilinear interpolation coefficients to remove redundant computation (ablation) | DCNv3 kernel · DCN runtime, 56 × 56 × 128 | module runtime (ms): 2.20 -> 2.18 | Helped |
| Use vectorized load/store to remove redundant memory instructions (ablation) | DCNv3 kernel · DCN runtime, 56 × 56 × 128 | module runtime (ms): 2.18 -> 1.28 | Helped |
Surprising findings
- Applying softmax to ConvNeXt depthwise convolution weights substantially slows convergence and lowers final accuracy (83.8 to 81.6 at 300 epochs in Table 1 for the softmax variant), suggesting bounded weights hurt expressiveness.
- Reducing per-channel threads to a per-group thread alone did not speed up the kernel; the speedup came mainly from vectorized loads and half precision.
Limitations
- Speed results are measured on a single GPU type (A100), and the paper notes header/decoder components in BEVFormer v2 are under-optimized.
- DCNv4 architectures were not re-tuned for the operator in ConvNeXt, ViT, and diffusion experiments, so hyperparameters may be suboptimal.
Experimental setup
Datasets: ImageNet-1K ImageNet-22K COCO ADE20K nuScenes
Benchmarks: ImageNet-1K COCO val2017 ADE20K nuScenes
Metrics: top-1 accuracy box AP mask AP mIoU NDS mAP FPS throughput runtime (ms) FID
Baselines: DCNv3 FlashAttention-2 Window attention DWConv Swin Transformer ConvNeXt InternImage ViT U-Net
Builds on: DCNv3 InternImage FlashAttention ConvNeXt U-ViT
Closest papers by meaning
- Deformable ConvNets V2: More Deformable, Better Results (CVPR 2019 · similarity 0.89)
- Deformable Kernels: Adapting Effective Receptive Fields for Object Deformation (ICLR 2020 · similarity 0.85)
- Vision-RWKV: Efficient and Scalable Visual Perception with RWKV-Like Architectures (ICLR 2025 · similarity 0.84)
- LaVin-DiT: Large Vision Diffusion Transformer (CVPR 2025 · similarity 0.84)
- KernelWarehouse: Rethinking the Design of Dynamic Convolution (ICML 2024 · similarity 0.84)
- Neural Residual Diffusion Models for Deep Scalable Vision Generation (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.