Research map / Large language models / Language, safety and interpretability
OPUS: Towards Efficient and Principled Data Selection in Large Language Model Pre-training in Every Iteration
Shaobo Wang, Xuan Ouyang, Tianyi Xu, Yuzheng Hu, Jialin Liu, Guo Chen, Tianyu Zhang, Junhao Zheng, Kexin Yang, Xingzhang Ren, Dayiheng Liu, Linfeng Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use OPUS instead of random selection | GPT-2 XL, Muon · FineWeb 30B | average accuracy: 40.29 -> 41.75 | Improved |
| use OPUS instead of random selection | GPT-2 Large, Muon · FineWeb 30B | average accuracy: 39.72 -> 40.25 | Improved |
| use OPUS instead of random selection | GPT-2 Large, AdamW · FineWeb 30B | average accuracy: 39.29 -> 41.43 | Improved |
| use OPUS instead of random selection | GPT-2 XL, AdamW · FineWeb 30B | average accuracy: 40.36 -> 41.01 | Improved |
| select with OPUS from score-3 pool instead of random from score-3 pool | GPT-2 XL, Muon · FineWeb-Edu 30B | average accuracy: 41.81 -> 44.99 | Improved |
| select with OPUS from score-3 pool instead of random from score-3 pool | GPT-2 Large, Muon · FineWeb-Edu 30B | average accuracy: 41.17 -> 41.72 | Improved |
| OPUS selecting from lower-quality score-3 pool instead of score 4+5 pool | GPT-2 XL, Muon · FineWeb-Edu 30B | average accuracy: 42.42 -> 44.99 | Improved |
| OPUS selecting from lower-quality score-3 pool instead of score 4+5 pool | GPT-2 Large, Muon · FineWeb-Edu 30B | average accuracy: 42.04 -> 41.72 | Hurt |
| use OPUS instead of random selection | GPT-2 XL · OOD benchmarks | average accuracy: 38.09 -> 40.07 | Improved |
| use OPUS instead of random selection | GPT-2 XL · BBH | exact match: 9.87 -> 11.02 | Improved |
| use OPUS instead of random selection | GPT-2 · FineWeb-Edu domain perplexity | average perplexity: 3.38 -> 3.35 | Improved |
| use OPUS instead of Random trained for 60B tokens (token budget) | GPT-2 XL · FineWeb-Edu validation loss | tokens to reach loss: 60B -> 17B | Improved |
Closest papers by meaning
- Data Selection via Optimal Control for Language Models (ICLR 2025 · similarity 0.90)
- Towards Scalable Data Diversification for Language Model Pretraining via Leverage Score Sampling (NeurIPS 2026 · similarity 0.89)
- Predictive Data Selection: The Data That Predicts Is the Data That Teaches (ICML 2025 · similarity 0.89)
- BLADE: Scalable Bi-level Adaptive Data Selection for LLM Training (NeurIPS 2026 · similarity 0.89)
- MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Models (NeurIPS 2024 · similarity 0.87)
- MuRating: A High Quality Data Selecting Approach to Multilingual Large Language Model Pretraining (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
