Research map / Architectures and efficiency / Pruning and efficient tuning
Big Transfer (BiT): General Visual Representation Learning
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, Neil Houlsby
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| pre-train on ImageNet-21k instead of ILSVRC-2012 | BiT ResNet152x4 · ILSVRC-2012 | top-1 accuracy: 81.30 -> 85.39 | Improved |
| pre-train on ImageNet-21k instead of ILSVRC-2012 | BiT ResNet152x4 · Flowers | top-1 accuracy: 89.89 -> 99.30 | Improved |
| pre-train on ImageNet-21k instead of ILSVRC-2012 | BiT ResNet152x4 · VTAB-1k | top-1 accuracy: 66.87 -> 70.64 | Improved |
| pre-train on ImageNet-21k instead of ILSVRC-2012 | BiT ResNet152x4 · Pets | top-1 accuracy: 93.97 -> 94.46 | Improved |
| pre-train on JFT-300M instead of ImageNet-21k | RetinaNet ResNet-101x3 · COCO-2017 val | AP: 43.2 -> 43.8 | Improved |
| add weight standardization to group normalization (ablation) | ResNet-50 · ILSVRC-2012 batch 4096 | top-1 accuracy: 70.2 -> 76.0 | Improved |
| replace batch norm with GN+WS (ablation) | ResNet-50 · VTAB-1k | top-1 accuracy: 67.72 -> 70.39 | Improved |
| add weight standardization to batch norm (ablation) | ResNet-50 · VTAB-1k | top-1 accuracy: 67.72 -> 66.78 | Hurt |
| add weight standardization to batch norm (ablation) | ResNet-50 · ILSVRC-2012 | top-1 accuracy: 75.6 -> 75.8 | No clear effect |
| tune hyperparameters per task with random search instead of BiT-HyperRule | BiT-L · VTAB-1k | top-1 accuracy: 76.29 -> 78.72 | Improved |
| scale pre-training to JFT-300M with small ResNet-50x1 model | ResNet-50x1 · downstream benchmarks | accuracy | Hurt |
| use lower weight decay in large-scale pre-training (ablation) | BiT pre-training · JFT-300M | validation error | Hurt |
Closest papers by meaning
- Scalable Transfer Learning with Expert Models (ICLR 2021 · similarity 0.91)
- Transferring Vision-Language Models for Visual Recognition: A Classifier Perspective (IJCV 2023 · similarity 0.90)
- On the Surprising Effectiveness of Attention Transfer for Vision Transformers (NeurIPS 2024 · similarity 0.88)
- Knowledge Transfer from Vision Foundation Models for Efficient Training of Small Task-specific Models (ICML 2024 · similarity 0.88)
- DTL: Disentangled Transfer Learning for Visual Recognition (AAAI 2024 · similarity 0.87)
- A Closer Look at the Few-Shot Adaptation of Large Vision-Language Models (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
