Research map / Label-efficient and robust learning / Recognition and long-tail
Data Selection for Language Models via Importance Resampling
Sang Michael Xie, Shibani Santurkar, Tengyu Ma, Percy Liang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| generative n-gram importance weights instead of fasttext discriminative (ablation) | DSIR · Continued pretraining avg F1 | F1: 83.15 -> 83.76 | Improved |
| generative n-gram weights instead of n-gram logistic regression discriminative (ablation) | DSIR · Continued pretraining avg F1 | F1: 83.19 -> 83.76 | Improved |
| add bigrams to unigram hashed features (ablation) | DSIR · Continued pretraining avg F1 | F1: 83.50 -> 83.76 | Improved |
| importance resampling instead of top-k selection (ablation) | DSIR · GLUE dev avg | accuracy: 81.38 -> 82.30 | Improved |
| DSIR vs random selection for general-domain pretraining | BERT-base MLM · GLUE dev avg | accuracy: 80.25 -> 82.30 | Improved |
| heuristic classification vs random selection from Pile | BERT-base MLM · GLUE dev avg | accuracy: 80.25 -> 79.85 | Hurt |
| DSIR vs manual curation (DAPT) for domain-specific continued pretraining | RoBERTa-base · Continued pretraining avg F1 | F1: 83.46 -> 83.76 | Improved |
| DSIR vs top-k heuristic classification | RoBERTa-base · Continued pretraining avg F1 | F1: 83.65 -> 83.76 | No clear effect |
| mismatched (worst) target distribution for pretraining data | DSIR · Downstream F1 across tasks | F1: about 6% average drop | Hurt |
Closest papers by meaning
- Towards Scalable Data Diversification for Language Model Pretraining via Leverage Score Sampling (NeurIPS 2026 · similarity 0.88)
- Harnessing Diversity for Important Data Selection in Pretraining Large Language Models (ICLR 2025 · similarity 0.88)
- Predictive Data Selection: The Data That Predicts Is the Data That Teaches (ICML 2025 · similarity 0.88)
- MuRating: A High Quality Data Selecting Approach to Multilingual Large Language Model Pretraining (NeurIPS 2025 · similarity 0.88)
- Get more for less: Principled Data Selection for Warming Up Fine-Tuning in LLMs (ICLR 2024 · similarity 0.87)
- Leveraging Importance Sampling to Detach Alignment Modules from Large Language Models (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
