AI Paper Atlas

Research map / Detection and segmentation / CNN features

Revitalizing CNN Attention via Transformers in Self-Supervised Visual Representation Learning

Chongjian GE, Youwei Liang, Yibing Song, Jianbo Jiao, Jue Wang, Ping Luo

NeurIPS 2021 · Detection and segmentation › CNN features · approach: Convolution-transformer backbones · 9 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Add transformer-guided attention supervision to the CNN stream (ablation)CARE · ImageNet, SSL linear evaluation, 100eptop-1 accuracy: 70.52 -> 72.06Improved
Set attention supervision weight λ above zero (ablation)CARE · ImageNet, SSL linear evaluation, 100eptop-1 accuracy: 70.52 -> 70.88; 70.52 -> 70.96; 70.52 -> 72.06Improved
Increase attention supervision weight from λ=100 to λ=250 (ablation)CARE · ImageNet, SSL linear evaluation, 100eptop-1 accuracy: 72.06 -> 72.00No clear effect
Increase transformer capacity from 2 to 5 attention blocks (ablation)CARE · ImageNet, SSL linear evaluation, 100eptop-1 accuracy: 71.08 -> 72.06Improved
Increase transformer capacity from 5 to 6 attention blocks (ablation)CARE · ImageNet, SSL linear evaluation, 100eptop-1 accuracy: 72.06 -> 69.37Hurt
Add learnable relative positional encoding instead of no positional encoding (ablation)CARE · ImageNet, SSL linear evaluation, 100eptop-1 accuracy: 69.49 -> 72.06Improved
Replace learnable relative encoding with sine-cosine absolute encoding (ablation)CARE · ImageNet, SSL linear evaluation, 100eptop-1 accuracy: 72.06 -> 66.68Hurt
Replace learnable relative encoding with learnable absolute encoding (ablation)CARE · ImageNet, SSL linear evaluation, 100eptop-1 accuracy: 72.06 -> 72.01No clear effect
Use parallel rather than sequential CNN/transformer training (ablation)CARE · ImageNet, SSL linear evaluation, 100eptop-1 accuracy: 69.32 -> 72.02Improved
Use CARE rather than BYOL pretrainingResNet-50 · ImageNet val, linear evaluation, 400eptop-1 accuracy: 73.2 -> 74.7Improved
Use CARE rather than BYOL pretrainingResNet-50 · ImageNet val, semi-supervised 1% labelstop-1 accuracy: 53.2 -> 60.0Improved
Use CARE rather than BYOL pretrainingResNet-50 · COCO detection, Mask R-CNN, 200ep pretrainingAPbb: 39.2 -> 39.4Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy