Research map / Label-efficient and robust learning / Recognition and long-tail
Revisiting Few-sample BERT Fine-tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use bias-corrected Adam instead of BERTAdam (ablation) | BERT-large fine-tuning · RTE | accuracy | Helped |
| Use bias-corrected Adam instead of BERTAdam (ablation) | BERT-large fine-tuning · MNLI | test accuracy: 87.0 \pm 0.4 -> 86.9 \pm 0.3 | No clear effect |
| Re-initialize the pooler and selected top Transformer blocks (ablation) | BERT-large fine-tuning · GLUE few-sample datasets | test performance: 69.5 \pm 2.5 -> 72.6 \pm 1.6; 90.8 \pm 1.3 -> 91.4 \pm 0.8; 89.0 \pm 0.6 -> 89.4 \pm 0.2; 63.0 \pm 1.5 -> 63.9 \pm 1.9 | Helped |
| Re-initialize more top layers (ablation) | BERT-large fine-tuning · RTE and MRPC | validation performance | Mixed |
| Train longer than 3 epochs (ablation) | BERT-large fine-tuning · Few-sample GLUE datasets | test performance: RTE: 69.5 \pm 2.5 -> 72.3 \pm 1.9; MRPC: 90.8 \pm 1.3 -> 90.5 \pm 1.5; STS-B: 89.0 \pm 0.6 -> 89.6 \pm 0.3; CoLA: 63.0 \ | Mixed |
| Transfer from an intermediate task (MNLI) | BERT-large fine-tuning · RTE | accuracy: 69.5 \pm 2.5 -> 81.8 \pm 1.7 | Helped |
| Apply layer-wise learning rate decay | BERT-large fine-tuning · GLUE few-sample datasets | test performance | Mixed |
| Apply Mixout regularization | BERT-large fine-tuning · GLUE few-sample datasets | test performance | Mixed |
| Apply pre-trained weight decay | BERT-large fine-tuning · GLUE few-sample datasets | test performance | No clear effect |
| Apply conventional weight decay | BERT-large fine-tuning · GLUE few-sample datasets | test performance | No clear effect |
| Use mixed precision instead of full precision (ablation) | BERT-large fine-tuning · RTE, MRPC, STS-B, CoLA | test performance | No clear effect |
Closest papers by meaning
- On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines (ICLR 2021 · similarity 0.91)
- The MultiBERTs: BERT Reproductions for Robustness Analysis (ICLR 2022 · similarity 0.88)
- IsoBN: Fine-Tuning BERT with Isotropic Batch Normalization (AAAI 2021 · similarity 0.87)
- Towards Understanding the Word Sensitivity of Attention Layers: A Study via Random Features (ICML 2024 · similarity 0.86)
- BERTs are Generative In-Context Learners (NeurIPS 2024 · similarity 0.85)
- Boosting Multi-Domain Fine-Tuning of Large Language Models through Evolving Interactions between Samples (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.