AI Paper Atlas

Research map / Architectures and efficiency / Pruning and efficient tuning

Big Transfer (BiT): General Visual Representation Learning

Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, Neil Houlsby

ECCV 2020 · Architectures and efficiency › Pruning and efficient tuning · approach: Efficient ViT adaptation · 1,221 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
pre-train on ImageNet-21k instead of ILSVRC-2012BiT ResNet152x4 · ILSVRC-2012top-1 accuracy: 81.30 -> 85.39Improved
pre-train on ImageNet-21k instead of ILSVRC-2012BiT ResNet152x4 · Flowerstop-1 accuracy: 89.89 -> 99.30Improved
pre-train on ImageNet-21k instead of ILSVRC-2012BiT ResNet152x4 · VTAB-1ktop-1 accuracy: 66.87 -> 70.64Improved
pre-train on ImageNet-21k instead of ILSVRC-2012BiT ResNet152x4 · Petstop-1 accuracy: 93.97 -> 94.46Improved
pre-train on JFT-300M instead of ImageNet-21kRetinaNet ResNet-101x3 · COCO-2017 valAP: 43.2 -> 43.8Improved
add weight standardization to group normalization (ablation)ResNet-50 · ILSVRC-2012 batch 4096top-1 accuracy: 70.2 -> 76.0Improved
replace batch norm with GN+WS (ablation)ResNet-50 · VTAB-1ktop-1 accuracy: 67.72 -> 70.39Improved
add weight standardization to batch norm (ablation)ResNet-50 · VTAB-1ktop-1 accuracy: 67.72 -> 66.78Hurt
add weight standardization to batch norm (ablation)ResNet-50 · ILSVRC-2012top-1 accuracy: 75.6 -> 75.8No clear effect
tune hyperparameters per task with random search instead of BiT-HyperRuleBiT-L · VTAB-1ktop-1 accuracy: 76.29 -> 78.72Improved
scale pre-training to JFT-300M with small ResNet-50x1 modelResNet-50x1 · downstream benchmarksaccuracyHurt
use lower weight decay in large-scale pre-training (ablation)BiT pre-training · JFT-300Mvalidation errorHurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy