AI Paper Atlas

Research map / Large language models / Language, safety and interpretability

Joint Selection for Large-Scale Pre-Training Data via Policy Gradient-based Mask Learning

Ziqing Fan, Yuqiao Xian, Yan Sun, Ke Shen, Li Shen

ICLR 2026 · Large language models › Language, safety and interpretability · approach: LLM unlearning and benchmarks · 3 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
select with joint quality+diversity mask learning (FineWeb-Mask) instead of raw FineWeb1.5B dense · 12-task averageaverage accuracy (%): 44.9 -> 48.1Improved
select with FineWeb-Mask instead of raw FineWeb7B MoE · 12-task averageaverage accuracy (%): 50.7 -> 52.6Improved
select with FineWeb-Mask instead of FineWeb-DCLM (best baseline)7B MoE · 12-task averageaverage accuracy (%): 52.2 -> 52.6Improved
select with FineWeb-Mask instead of FineWebPro (best baseline)1.5B dense · 12-task averageaverage accuracy (%): 47.2 -> 48.1Improved
diversity-only selection (Semdedup) instead of raw FineWeb1.5B dense · 12-task averageaverage accuracy (%): 44.9 -> 43.8Hurt
quality-only selection (FineWeb-Edu) instead of raw FineWeb1.5B dense · 12-task averageaverage accuracy (%): 44.9 -> 46.8Improved
add quality-aware pruning (ablation)DiSF mask selection · 12-task averageaverage accuracy (%): 44.3 -> 45.0Improved
add quality-aware initialization on top of pruning (ablation)DiSF mask selection · 12-task averageaverage accuracy (%): 45.0 -> 45.1Improved
add DiSF diversity to joint objective (ablation)DataMask joint learning · 12-task averageaverage accuracy (%)Hurt
add Pair-wise Similarity or Facility Location to quality score (ablation)DataMask joint learning · 12-task average (1.5B, 400B tokens)average accuracy (%)Improved
use FineWeb-Mask instead of FineWeb-Edu for continual pre-training plus SFT7B MoE · 12-task averageaverage accuracy (%): 59.1 -> 60.2Improved
use too small group number G (ablation)DataMask · optimized diversity valueoptimized scoreHurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy