Research map / Architectures and efficiency / Attention and transformers
ViT-CoMer: Vision Transformer with Convolutional Multi-scale Feature Interaction for Dense Predictions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ViT-CoMer, a pre-training-free plain ViT backbone augmented with a parallel CNN branch (multi-receptive field pyramid plus bidirectional CNN-Transformer fusion) for dense prediction.
Problem
Plain ViTs perform poorly on dense prediction because they lack inner-patch local interaction and multi-scale feature diversity, while existing vision-specific transformers require costly re-pretraining and cannot directly reuse advanced ViT pretrained weights.
Approach
Keep the plain ViT unchanged so it can load open-source pretrained weights, and add a CNN branch with a multi-receptive-field feature pyramid (MRFP) plus a CNN-Transformer bidirectional fusion interaction (CTI) module at each of N stages to inject multi-scale convolutional features into the ViT.
Key results
- ViT-CoMer-L with advanced pretraining reaches 64.3 AP on COCO val2017 (Co-DETR with BEiTv2) without extra detection data.
- ViT-CoMer-L reaches 62.1 mIoU on ADE20K val with Mask2Former and BEiTv2 initialization.
- Mask R-CNN 1x: ViT-CoMer-S improves over plain ViT-S by +5.6 box AP and +3.4 mask AP (45.8 / 40.5).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ViT-CoMer-L‡ instead of ViT-Adapter-L† | COCO val2017 Mask R-CNN 3× · COCO val2017 | box AP: 52.1 -> 55.9 | Helped |
| Use ViT-CoMer-L instead of HTC++ with ViT-Adapter-L | Co-DETR · COCO val2017 | box AP: 62.6 -> 64.3 | Helped |
| Use ViT-CoMer-L instead of ViT-Adapter-L | Mask2Former · ADE20K val | mIoU: 61.5 -> 62.1 | Helped |
| Add MRFP to plain ViT-S (ablation) | ViT-S · COCO val2017, Mask R-CNN 1× | box AP: 40.2 -> 41.5 | Helped |
| Add MRFP to plain ViT-S (ablation) | ViT-S · COCO val2017, Mask R-CNN 1× | mask AP: 37.1 -> 38.2 | Helped |
| Add CTI feature injection into the ViT branch (ablation) | ViT-S + MRFP · COCO val2017, Mask R-CNN 1× | box AP: 41.5 -> 43.3 | Helped |
| Add CTI feature injection into the CNN branch (ablation) | ViT-S + MRFP + CTI (to V) · COCO val2017, Mask R-CNN 1× | box AP: 43.3 -> 45.8 | Helped |
| Use 2 bidirectional fusion interaction modules instead of none (ablation) | ViT-S · COCO val2017, Mask R-CNN 1× | box AP: 40.2 -> 45.1 | Helped |
| Use 6 rather than 4 bidirectional fusion interaction modules (ablation) | ViT-CoMer-S · COCO val2017, Mask R-CNN 1× | box AP: 45.8 -> 45.6 | Hurt |
| Use kernel sizes 3 and 5 rather than size 3 alone in MRFP (ablation) | ViT-CoMer-S · COCO val2017, Mask R-CNN 1× | box AP: 45.7 -> 45.8 | Helped |
| Add kernel size 7 to MRFP (ablation) | ViT-CoMer-S with kernel sizes 3 and 5 · COCO val2017, Mask R-CNN 1× | box AP: 45.8 -> 45.5 | Hurt |
| Add kernel size 9 to MRFP (ablation) | ViT-CoMer-S with kernel sizes 3, 5, and 7 · COCO val2017, Mask R-CNN 1× | box AP: 45.5 -> 45.4 | Hurt |
Limitations
- Gains over Swin-style vision-specific backbones are smaller at comparable size; the Swin scalability result shows smaller improvement.
- Top results rely on DINOv2 or BEiTv2 pretraining and Co-DETR/Mask2Former heads, so the headline numbers are not from the backbone alone.
Experimental setup
Datasets: COCO ADE20K ImageNet-1K ImageNet-22K
Benchmarks: COCO val2017 ADE20K val
Metrics: box AP mask AP mIoU
Baselines: ViTDet ViT-Adapter Swin PVT InternImage MixFormer Co-DETR Mask2Former
Builds on: ViT ViTDet ViT-Adapter Mask R-CNN UperNet Mask2Former Co-DETR
Closest papers by meaning
- Vision Transformer Adapter for Dense Predictions (ICLR 2023 · similarity 0.91)
- Memory Efficient Transformer Adapter for Dense Predictions (ICLR 2025 · similarity 0.87)
- Bi-ViT: Pushing the Limit of Vision Transformer Quantization (AAAI 2024 · similarity 0.86)
- Slicing Vision Transformer for Flexible Inference (NeurIPS 2024 · similarity 0.86)
- LaVin-DiT: Large Vision Diffusion Transformer (CVPR 2025 · similarity 0.86)
- LF-ViT: Reducing Spatial Redundancy in Vision Transformer for Efficient Image Recognition (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.