Research map / Architectures and efficiency / RNNs and normalization
The MultiBERTs: BERT Reproductions for Robustness Analysis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Continue pre-training each base checkpoint with counterfactual data augmentation for 50K steps | BERT-based coreference systems · Winogender | Bias Corr. (r): 0.423 -> 0.261; -0.162; p = 0.001 | Helped |
| Continue pre-training each base checkpoint with counterfactual data augmentation for 50K steps | BERT-based coreference systems · Winogender | accuracy: 0.626 -> 0.623; -0.004; p = 0.210 | No clear effect |
| Train from scratch with counterfactual data augmentation rather than continue pre-training with it | CDA-incr BERT-based coreference systems · Winogender | Bias Corr. (r): 0.256 -> 0.192; -0.064; p = 0.132 | No clear effect |
| Train from scratch with counterfactual data augmentation rather than continue pre-training with it | CDA-incr BERT-based coreference systems · Winogender | accuracy: 0.623 -> 0.622; -0.001; p = 0.416 | No clear effect |
| Bootstrap over seeds only, holding occupations fixed (ablation) | CDA-full vs. CDA-incr comparison · Winogender | p-value for Bias Corr. (r): 0.132 -> 0.005 | Mixed |
| Bootstrap over examples only, holding pre-training seeds fixed (ablation) | CDA-full vs. CDA-incr comparison · Winogender | p-value for Bias Corr. (r): 0.132 -> 0.053 | Mixed |
Closest papers by meaning
- Revisiting Few-sample BERT Fine-tuning (ICLR 2021 · similarity 0.88)
- On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines (ICLR 2021 · similarity 0.87)
- PolyPythias: Stability and Outliers across Fifty Language Model Pre-Training Runs (ICLR 2025 · similarity 0.85)
- Underspecification in Language Modeling Tasks: A Causality-Informed Study of Gendered Pronoun Resolution (AAAI 2024 · similarity 0.84)
- An Empirical Study on Noisy Data and LLM Pretraining Loss Divergence (NeurIPS 2026 · similarity 0.84)
- Data Mixture Inference Attack: BPE Tokenizers Reveal Training Data Compositions (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.