Research map / Architectures and efficiency / RNNs and normalization
Fast-ELECTRA for Efficient Pre-training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Fast-ELECTRA replaces ELECTRA's jointly trained auxiliary generator with an existing pretrained MLM whose output is temperature-smoothed on an exponentially decaying schedule, cutting auxiliary compute and memory while matching GLUE performance.
Problem
ELECTRA-style pre-training jointly trains an auxiliary masked language model that is discarded after pre-training, wasting substantial compute and memory and making hyper-parameter balancing difficult.
Approach
Use an existing (frozen, inference-only) language model as the replaced-token generator, smoothing its output distribution with temperature scaling whose temperature decays exponentially over training to form a difficulty curriculum that substitutes for joint training.
Key results
- Fast-ELECTRA reaches GLUE average 88.7 in the Base setting, comparable to METRO (88.6) and above ELECTRA (85.5) on the dev set.
- Auxiliary model computation is reduced to about 0.33 of the original (132.9 vs 398.6 GFLOPs), and total computation by about 27% (724.8 vs 990.5 GFLOPs) in Base.
- Auxiliary memory drops from 7.0 GB to 0.25 GB in Base, and training remains stable at learning rate 2e-3 where original ELECTRA diverges.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace joint generator training with an existing auxiliary model and scheduled temperature scaling | Fast-ELECTRA vs. Original ELECTRA · GLUE dev, Base | Average score: 88.5 -> 88.7 | Helped |
| replace joint generator training with an existing auxiliary model and scheduled temperature scaling | Fast-ELECTRA vs. METRO ReImp · GLUE dev, Large | Average score: 89.2 -> 89.3 | Helped |
| use an inference-only auxiliary model instead of a jointly trained one | ELECTRA, Base · Base | Total computation (GFLOPs): 990.5 -> 724.8 | Helped |
| use an inference-only auxiliary model instead of a jointly trained one | ELECTRA, Base · Base | Total memory (GB): 30.0 -> 23.3 | Helped |
| use an inference-only auxiliary model instead of a jointly trained one | ELECTRA, Large · Large | Total computation (GFLOPs): 2061.6 -> 1625.6 | Helped |
| use an inference-only auxiliary model instead of a jointly trained one | ELECTRA, Large · Large | Total memory (GB): 74.6 -> 60.6 | Helped |
| replace the uniform replacement distribution with term-frequency replacements (ablation) | auxiliary-model-free RTD · MNLI | Accuracy: Diverged -> 85.7 | Helped |
| interpolate uniform and smoothed one-hot replacement distributions (ablation) | auxiliary-model-free RTD · MNLI | Accuracy: 83.8 -> 85.3 | Helped |
| interpolate term frequency and smoothed one-hot replacement distributions (ablation) | auxiliary-model-free RTD · MNLI | Accuracy: 85.7 -> 84.9 | Hurt |
| use an auxiliary-model-free replacement distribution instead of Fast-ELECTRA's auxiliary-model-based replacements (ablation) | Fast-ELECTRA · MNLI | Accuracy: 89.1 -> 85.7 | Hurt |
| apply scheduled temperature scaling to the fixed auxiliary model (ablation) | fixed-auxiliary pre-training · MNLI, 2K updates | Accuracy: 80% -> 85% | Helped |
| apply temperature scaling to the fixed auxiliary model (ablation) | fixed-auxiliary pre-training · MNLI, upon convergence | Accuracy: 88.4% vs. 89.1% | Helped |
Surprising findings
- Pre-training with a fixed auxiliary model for over 10K updates yields zero replaced-token detection accuracy before it starts converging.
- Pre-training with term frequency alone as the replaced-token distribution (no auxiliary model) reaches about 85.7% MNLI accuracy, and uniform distribution diverges early.
Limitations
- Experiments limited to BERT-base equivalent and BERT-large scale; scaling to large language models untested by authors.
- Training stability evaluated only at standard model size due to compute constraints.
- Reported GLUE baselines partly taken from papers with differing training budgets.
Experimental setup
Datasets: Wikipedia BookCorpus GLUE MNLI
Benchmarks: GLUE MNLI
Metrics: accuracy Matthews correlation Spearman correlation GLUE average score FLOPs per batch peak memory (GB) seconds per update
Baselines: ELECTRA METRO RoBERTa BERT XLNet DeBERTa COCO-LM AMOS DeBERTaV3 TUPE MC-BERT
Builds on: ELECTRA METRO BERT simulated annealing
Closest papers by meaning
- Progressive Residual Warmup for Language Model Pretraining (NeurIPS 2026 · similarity 0.87)
- Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training (ICLR 2024 · similarity 0.86)
- Efficient Pre-Training with Token Superposition (NeurIPS 2026 · similarity 0.86)
- Beyond Next Token Prediction: Patch-Level Training for Large Language Models (ICLR 2025 · similarity 0.85)
- APT: Adaptive Pruning and Tuning Pretrained Language Models for Efficient Training and Inference (ICML 2024 · similarity 0.85)
- Cross-model Control: Improving Multiple Large Language Models in One-time Training (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.