Research map / Large language models / Language, safety and interpretability
Group-Level Data Selection for Efficient Pretraining
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Group-MATES, which trains a relational data influence model on sampled trajectories with oracle influences and uses influence-aware clustering to select pretraining data at the group level.
Problem
Pretraining data selection methods score points individually, ignoring interactions among data in a group, while exact group-level selection is combinatorially infeasible.
Approach
Parameterize greedy group selection with a relational data influence model that weights each candidate's individual influence prediction by cosine-similarity relationships to already-selected points, trained on rollout trajectories with bootstrapped tail sampling, and made efficient by clustering the pool with relationship-aware (influence-aware) clustering.
Key results
- Group-MATES achieves 3.5%-9.4% relative Core gains over random selection across DCLM 400M-4x, 1B-1x and 3B-1x, with 23.36 vs 21.36 Core at 400M-4x.
- At 1B-1x Group-MATES reaches 0.30747 Core versus 0.30288 for MATES and 0.29456 for random.
- Group-MATES reduces tokens needed to reach a given Core score by up to 1.75x relative to random selection.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Group-MATES instead of random selection | DCLM 400M-4x pretraining · DCLM 400M-4x | Core score: 3.5%-9.4% relative improvements in Core scores across all three setups | Helped |
| Use Group-MATES instead of random selection | DCLM 1B-1x pretraining · DCLM 1B-1x | Core score: 3.5%-9.4% relative improvements in Core scores across all three setups | Helped |
| Use Group-MATES instead of random selection | DCLM 3B-1x pretraining · DCLM 3B-1x | Core score: 3.5%-9.4% relative improvements in Core scores across all three setups | Helped |
| Add relationship weights to the selection model (ablation) | Group-MATES · DCLM 400M-4x | Core score: 0.22737 -> 0.23362 | Helped |
| Add bootstrapping to the rollout policy (ablation) | Group-MATES · DCLM 400M-4x | Core score: 0.22924 -> 0.23362 | Helped |
| Use influence-aware rather than semantic clusters (ablation) | Group-MATES · DCLM 400M-4x | Core score: 0.23042 -> 0.23362 | Helped |
| Increase rollout length from 2 to 10 (ablation) | Group-MATES · DCLM 1B-1x | Core score: 0.30685 -> 0.30747 | Helped |
| Increase rollout length from 10 to 20 (ablation) | Group-MATES · DCLM 1B-1x | Core score: 0.30747 -> 0.30262 | Hurt |
| Use a 25% rather than 10% selection ratio (ablation) | Group-MATES · DCLM 400M-4x | Core score: 0.22743 -> 0.23212 | Helped |
| Use a 50% rather than 25% selection ratio (ablation) | Group-MATES · DCLM 400M-4x | Core score: 0.23212 -> 0.23362 | No clear effect |
| Replace BGE backbone with BERT-base (ablation) | Relational data influence model · Oracle influence approximation | Spearman correlation | Hurt |
| Use dot product instead of cosine similarity (ablation) | Relational data influence model · Oracle influence approximation | Spearman correlation | Hurt |
Surprising findings
- Removing relationship weights cuts the Spearman correlation of the influence model to near zero, showing individual-only influence models fail to approximate group-level oracle influence.
- Individual-influence selection (MATES) diverges from greedy group selection after about 100 selected points, and its overlap with group selection approaches random.
Limitations
- Selection overhead (oracle collection and inference) adds FLOPs, and the gains are measured on a single benchmark family (DCLM).
- Group-level selection gains are evaluated only with short decay-stage experiments in the analysis of greedy selection.
- Relationship weights are based on embedding cosine similarity, which may not capture all interactions.
Experimental setup
Datasets: DCLM FLAN
Benchmarks: DCLM 22-task Core evaluation SciQ ARC-E ARC-C LogiQA OBQA BoolQ HellaSwag PIQA WinoGrande
Metrics: Core score (centered accuracy) relative gain over random Spearman correlation FLOPs-to-performance
Baselines: Random selection Edu Classifier WebOrganizer MATES Quad DSIR SemDeDup DsDm QuRating
Builds on: MATES Quad DCLM BGE
Closest papers by meaning
- MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Models (NeurIPS 2024 · similarity 0.95)
- Harnessing Diversity for Important Data Selection in Pretraining Large Language Models (ICLR 2025 · similarity 0.91)
- Predictive Data Selection: The Data That Predicts Is the Data That Teaches (ICML 2025 · similarity 0.89)
- Improving Pretraining Data Using Perplexity Correlations (ICLR 2025 · similarity 0.88)
- Efficient Data Selection at Scale via Influence Distillation (NeurIPS 2025 · similarity 0.88)
- Get more for less: Principled Data Selection for Warming Up Fine-Tuning in LLMs (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.