Research map / Large language models / Instruction and fine-tuning
On the Transferability of Pre-trained Language Models: A Study from Artificial Datasets
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add explicit paired-token dependencies to pre-training sequences | RoBERTa-medium · GLUE Avg | relative improvement vs Scratch: Uni-gram +1.3; Flat-2 +7.2 | Helped |
| increase maximum paired-token dependency length from 2 to 4 (ablation) | Flat Parentheses pre-training · GLUE Avg | relative improvement vs Scratch: +7.2 -> +10.7 | Helped |
| increase maximum paired-token dependency length from 4 to 6 (ablation) | Flat Parentheses pre-training · GLUE Avg | relative improvement vs Scratch: +10.7 -> +13.5 | Helped |
| use nested rather than flat paired-token dependencies (ablation) | RoBERTa-medium pre-training · GLUE Avg | relative improvement vs Scratch: +13.5 vs +12.0 | No clear effect |
| add implicit local dependencies by pre-training on shuffled token blocks | RoBERTa-medium · GLUE Avg | relative improvement vs Scratch: Uni-gram +1.3; Shuff.-4 +4.7 | Helped |
| increase shuffled block size from 4 to 6 (ablation) | Shuffle-N pre-training · GLUE Avg | relative improvement vs Scratch: +4.7 -> +6.4 | Helped |
| increase shuffled block size from 6 to 64 (ablation) | Shuffle-N pre-training · STS-B | Spearman correlation: +10.3 -> +49.7 | Helped |
| match the English unigram or bigram distribution without token dependencies | RoBERTa-medium · GLUE Avg | relative improvement vs Scratch: Uniform -0.2; Uni-gram +1.3; Bi-gram +2.2 | No clear effect |
| use English pre-training rather than artificial pre-training | RoBERTa-medium · GLUE Avg | relative improvement vs Scratch: Pre. En +20.5; Flat-128 +14.5 | Helped |
| fine-tune artificial-pretrained models on QNLI and evaluate on identical-question adversarial inputs | Nest.Par. and Shuffle-4 · QNLI-Adv | accuracy: Nest.Par. 1.0; Shuffle-4 1.1 | Hurt |
| randomly initialize the word embedding layer instead of retaining its pre-trained weights (ablation) | RoBERTa-medium with artificial pre-training · GLUE | performance | Hurt |
Closest papers by meaning
- DEPT: Decoupled Embeddings for Pre-training Language Models (ICLR 2025 · similarity 0.89)
- On Linear Representations and Pretraining Data Frequency in Language Models (ICLR 2025 · similarity 0.88)
- Seeking Neural Nuggets: Knowledge Transfer in Large Language Models from a Parametric Perspective (ICLR 2024 · similarity 0.88)
- Scalable Influence and Fact Tracing for Large Language Model Pretraining (ICLR 2025 · similarity 0.88)
- Once Read is Enough: Domain-specific Pretraining-free Language Models with Cluster-guided Sparse Experts for Long-tail Domain Knowledge (NeurIPS 2024 · similarity 0.88)
- An Empirical Study on Noisy Data and LLM Pretraining Loss Divergence (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.