Research map / Large language models / Language, safety and interpretability
Curriculum-Guided Layer Scaling for Language Model Pretraining
Karanpartap Singh, Neil Band, Ehsan Adeli
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the freezing phase when introducing new layers (ablation) | CGLS · Llama-3.2-1B (2.5B tokens) | average accuracy: 37.46 -> 39.08 | Improved |
| add the freezing phase when introducing new layers (ablation) | CGLS · Llama-3.2-1B (2.5B tokens) | stage 1 mixture perplexity: 30.84 -> 24.50 | Improved |
| add synchronized (easy to hard) curriculum ordering instead of reversed (ablation) | CGLS · Llama-3.2-1B (2.5B tokens) | average accuracy: 32.52 -> 39.08 | Improved |
| use FineWeb difficulty classifier instead of DistilBERT classifier (ablation) | CGLS · Llama-3.2-1B (2.5B tokens) | average accuracy: 39.08 -> 39.06 | No clear effect |
| add layer scaling and data curriculum together (CGLS) versus layer scaling only (ablation) | Layer Scaling Only · Llama-3.2-1B (2.5B tokens) | average accuracy: 34.24% -> 39.08% | Improved |
| add data curriculum to the standard baseline (ablation) | Baseline (Randomized) · Llama-3.2-1B (2.5B tokens) | average accuracy: 37.38% -> 37.33% | No clear effect |
| add layer scaling with curriculum (CGLS) versus randomized baseline | Baseline (Randomized) · GPT-2-Small (2.5B tokens) | average accuracy: 35.13% -> 36.61% | Improved |
| use CGLS instead of randomized baseline | Baseline (Randomized) · Llama-3.2-1B (2.5B tokens) | The Pile perplexity: 20.83 -> 25.34 | Hurt |
| use CGLS instead of randomized baseline at larger scale | Baseline (Randomized) · Llama-3.2-1B (20B tokens) | average accuracy: 38.41% -> 42.30% | Improved |
| use CGLS instead of randomized baseline at larger scale | Baseline (Randomized) · Llama-3.2-1B (20B tokens) | The Pile perplexity: 26.33 -> 22.99 | Improved |
| order data from general text to code with growing model (CGLS) | Baseline (Randomized mix) · HumanEval (DCLM to StarCoder2) | pass@8: 3.56% -> 6.67% | Improved |
| use CGLS instead of baseline across three seeds | Baseline · Llama-3.2-1B (2.5B tokens, replication) | average accuracy: 33.26 ± 0.24% -> 35.94 ± 1.60% | Improved |
Closest papers by meaning
- Webscale-RL: Automated Data Pipeline for Scaling RL Data to Pretraining Levels (ICLR 2026 · similarity 0.89)
- MiniPLM: Knowledge Distillation for Pre-training Language Models (ICLR 2025 · similarity 0.89)
- Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning (ICLR 2024 · similarity 0.89)
- Data Engineering for Scaling Language Models to 128K Context (ICML 2024 · similarity 0.88)
- Exploring Visual Pretraining for Learning Language Intelligence (CVPR 2026 · similarity 0.88)
- Task-Adaptive Pretrained Language Models via Clustered-Importance Sampling (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
