Research map / Large language models / Language, safety and interpretability
Efficient Training of Language Models using Few-Shot Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| gradual stacking (post-stack, equal steps) instead of baseline full-depth training | BERT-BASE pretraining · BERT pretraining | speedup: 1.50x vs 1x | Helped |
| gradual stacking instead of baseline | BERT-LARGE pretraining · BERT pretraining | speedup: 1.64x vs 1x | Helped |
| gradual stacking instead of progressive stacking | BERT-BASE and BERT-LARGE pretraining · BERT pretraining | pretrain loss: 1.689 vs 1.737 (base); 1.384 vs 1.403 (large) | Helped |
| gradual stacking with equal steps allocation | BERT-BASE and BERT-LARGE · downstream MNLI and SQuAD2.0 | accuracy / F1: MNLI +0.0056 (base), +0.0084 (large); SQuAD2.0 -0.0010 (base), +0.0037 (large) | Mixed |
| single-shot stacking instead of gradual stacking (ablation) | BERT-BASE and BERT-LARGE pretraining · BERT pretraining | pretrain loss: e.g. 1.849 (base) and 1.402 (large) vs 1.689 and 1.384 for gradual | Hurt |
| inverse proportional steps allocation instead of equal allocation (ablation) | gradual stacking · BERT pretraining | speedup vs pretrain loss: BERT-LARGE 2.11x speedup but loss 1.435 vs 1.384; BERT-BASE loss 1.801 vs 1.689 | Mixed |
| larger stack size (4 vs 1) in gradual stacking (ablation) | BERT-LARGE gradual stacking · BERT pretraining | pretrain loss: 1.384 (stack 4) vs 1.615 (stack 1); speedup drops 1.81x to 1.64x | Helped |
| post-stack instead of pre-stack (ablation) | gradual stacking BERT-BASE · BERT pretraining | pretrain loss: 1.689 vs 1.731 at same 1.50x speedup | Helped |
| varying sequence length across stages (128, 256, 512) (ablation) | gradual stacking BERT-LARGE · BERT pretraining | speedup: 2.44x vs 1.64x, with loss 1.429 vs 1.384 | Helped |
| independent stacking (two separately trained 6-layer models) instead of gradual stacking (ablation) | BERT-BASE two-stage training · BERT pretraining (Figure 1) | loss: qualitative, no numbers in text | Helped |
Closest papers by meaning
- Optimization Inspired Few-Shot Adaptation for Large Language Models (NeurIPS 2025 · similarity 0.88)
- Many-Shot In-Context Learning (NeurIPS 2024 · similarity 0.88)
- Language Models as Zero-shot Lossless Gradient Compressors: Towards General Neural Parameter Prior Models (NeurIPS 2024 · similarity 0.87)
- Pre-trained Vision and Language Transformers Are Few-Shot Incremental Learners (CVPR 2024 · similarity 0.87)
- Large Language Models are Good Prompt Learners for Low-Shot Image Classification (CVPR 2024 · similarity 0.87)
- Pruning Large Language Models with Semi-Structural Adaptive Sparse Training (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.