Research map / Architectures and efficiency / RNNs and normalization
Exploring the Limits of Large Scale Pre-training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase pre-training head weight decay from 0.0 to 5.0 (ablation) | ViT-B/32 · VTAB-1K CIFAR-100 | accuracy: 0.51 -> 0.79 | Helped |
| Increase pre-training head weight decay from 0.0 to 5.0 (ablation) | ViT-B/32 · VTAB-1K SUN397 | accuracy: 0.42 -> 0.65 | Helped |
| Increase pre-training head weight decay from 0.0 to 5.0 (ablation) | ViT-B/32 · VTAB-1K CLEVR-count | accuracy: 0.72 -> 0.42 | Hurt |
| Increase pre-training head weight decay from 0.0 to 5.0 (ablation) | ViT-B/32 · VTAB-1K dSprites-orientation | accuracy: 0.58 -> 0.58 | No clear effect |
| Increase pre-training head weight decay from 0.0 to 5.0 (ablation) | ViT-B/32 · VTAB-1K | accuracy: VTAB-ALL: 0.69 -> 0.68 | Mixed |
| Increase upstream pre-training accuracy through scaling | Vision Transformer, MLP-Mixer, and ResNet pre-training · Downstream image recognition tasks | accuracy | Mixed |
| Use representations from lower layers instead of the pre-logit layer (ablation) | Pre-trained model representations · ColHist | accuracy | Helped |
| Increase head weight decay beyond the upstream-optimal value (ablation) | Pre-training head · JFT upstream task versus selected downstream tasks | accuracy | Mixed |
| Decrease the head learning rate relative to the rest of the network (ablation) | Pre-training head · JFT upstream task and downstream tasks | accuracy | Mixed |
| Increase the number of downstream few-shot examples (ablation) | Frozen pre-trained representations and linear classifier · Downstream few-shot tasks | irreducible error | Helped |
| Fit the power law to the convex hull rather than all data points (ablation) | Downstream-versus-upstream accuracy prediction · Held-out high-upstream-accuracy points | prediction error | Helped |
| Scale model size, upstream data size, or training epochs (ablation) | Controlled ViT pre-training · Downstream few-shot image recognition | accuracy | Mixed |
Closest papers by meaning
- On the Surprising Effectiveness of Large Learning Rates under Standard Width Scaling (NeurIPS 2025 · similarity 0.86)
- Learning What to Predict: Downstream-Guided Task Design for Continued Pretraining (NeurIPS 2026 · similarity 0.85)
- Understanding and Mitigating the Label Noise in Pre-training on Downstream Tasks (ICLR 2024 · similarity 0.85)
- On the Surprising Effectiveness of Attention Transfer for Vision Transformers (NeurIPS 2024 · similarity 0.85)
- Scalable Pre-training of Large Autoregressive Image Models (ICML 2024 · similarity 0.85)
- Initialization Matters for Adversarial Transfer Learning (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.