Research map / Detection and segmentation / CNN features
Revitalizing CNN Attention via Transformers in Self-Supervised Visual Representation Learning
Chongjian GE, Youwei Liang, Yibing Song, Jianbo Jiao, Jue Wang, Ping Luo
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add transformer-guided attention supervision to the CNN stream (ablation) | CARE · ImageNet, SSL linear evaluation, 100ep | top-1 accuracy: 70.52 -> 72.06 | Improved |
| Set attention supervision weight λ above zero (ablation) | CARE · ImageNet, SSL linear evaluation, 100ep | top-1 accuracy: 70.52 -> 70.88; 70.52 -> 70.96; 70.52 -> 72.06 | Improved |
| Increase attention supervision weight from λ=100 to λ=250 (ablation) | CARE · ImageNet, SSL linear evaluation, 100ep | top-1 accuracy: 72.06 -> 72.00 | No clear effect |
| Increase transformer capacity from 2 to 5 attention blocks (ablation) | CARE · ImageNet, SSL linear evaluation, 100ep | top-1 accuracy: 71.08 -> 72.06 | Improved |
| Increase transformer capacity from 5 to 6 attention blocks (ablation) | CARE · ImageNet, SSL linear evaluation, 100ep | top-1 accuracy: 72.06 -> 69.37 | Hurt |
| Add learnable relative positional encoding instead of no positional encoding (ablation) | CARE · ImageNet, SSL linear evaluation, 100ep | top-1 accuracy: 69.49 -> 72.06 | Improved |
| Replace learnable relative encoding with sine-cosine absolute encoding (ablation) | CARE · ImageNet, SSL linear evaluation, 100ep | top-1 accuracy: 72.06 -> 66.68 | Hurt |
| Replace learnable relative encoding with learnable absolute encoding (ablation) | CARE · ImageNet, SSL linear evaluation, 100ep | top-1 accuracy: 72.06 -> 72.01 | No clear effect |
| Use parallel rather than sequential CNN/transformer training (ablation) | CARE · ImageNet, SSL linear evaluation, 100ep | top-1 accuracy: 69.32 -> 72.02 | Improved |
| Use CARE rather than BYOL pretraining | ResNet-50 · ImageNet val, linear evaluation, 400ep | top-1 accuracy: 73.2 -> 74.7 | Improved |
| Use CARE rather than BYOL pretraining | ResNet-50 · ImageNet val, semi-supervised 1% labels | top-1 accuracy: 53.2 -> 60.0 | Improved |
| Use CARE rather than BYOL pretraining | ResNet-50 · COCO detection, Mask R-CNN, 200ep pretraining | APbb: 39.2 -> 39.4 | Improved |
Closest papers by meaning
- A Closer Look at Self-Supervised Lightweight Vision Transformers (ICML 2023 · similarity 0.87)
- Efficient Self-supervised Vision Transformers for Representation Learning (ICLR 2022 · similarity 0.87)
- Vision Transformers Need Registers (ICLR 2024 · similarity 0.86)
- Harnessing small projectors and multiple views for efficient vision pretraining (NeurIPS 2024 · similarity 0.86)
- On the Surprising Effectiveness of Attention Transfer for Vision Transformers (NeurIPS 2024 · similarity 0.85)
- Skip-Attention: Improving Vision Transformers by Paying Less Attention (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
