Research map / Architectures and efficiency / RNNs and normalization
Target-Oriented Pretraining Data Selection via Neuron-Activated Graph
Zijun Wang, Haoqin Tu, Weidong Zhou, Yiyang Zhou, Xiaohuan Zhou, Bingni Zhang, Weiguo Feng, Taifeng Wang, Cihang Xie, Fengze Liu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| select pretraining data by NAG similarity instead of random sampling | 1.2B LM pretraining · six-benchmark average, single-target | average accuracy: 44.9% -> 49.8% | Improved |
| use NAG instead of BETR embedding similarity | target-oriented data selection · single-target average | average accuracy: 48.8% -> 49.8% | Improved |
| use NAG instead of BETR under naive multi-target mixture | target-oriented data selection · multi-target average | average accuracy: 44.4% -> 48.5% | Improved |
| restrict NAG to the last layer only (ablation) | NAG (Qwen3-1.7B) · six-benchmark average | average accuracy: 49.8% -> 45.7% | Hurt |
| use UP-projection neurons instead of K or DOWN projection neurons (ablation) | NAG · HellaSwag | accuracy: 56.69 (K), 57.99 (DOWN) -> 60.55 (UP) | Improved |
| deactivate NAG-selected neurons (0.12% of all) in the LLM (ablation) | Qwen3-1.7B-Base · six-benchmark average | average accuracy: 60.6% -> 37.1% | Hurt |
| deactivate an equal number of random neurons (ablation) | Qwen3-1.7B-Base · six-benchmark average | average accuracy: 60.6% -> 60.4% | No clear effect |
| add NAG ranking to FineWeb-Edu quality signal | FineWeb-Edu · single-target average | average accuracy: 47.4% -> 49.6% | Improved |
| lower filtering rate rf (keep top 5% instead of 20%) with NAG (ablation) | NAG ranking · HellaSwag | accuracy: 60.5% -> 62.3% | Improved |
| lower filtering rate rf for random selection (ablation) | random sampling · HellaSwag | accuracy: 51.6% -> 50.9% | Hurt |
| increase layerwise neuron ratio from 0.3% to 2.1% (ablation) | NAG (Qwen3-1.7B) · HellaSwag | accuracy | No clear effect |
| use a larger backbone (Qwen3 1.7B/4B/8B) for NAG (ablation) | NAG · HellaSwag | accuracy | Improved |
Closest papers by meaning
- Task-Specific Data Selection for Instruction Tuning via Monosemantic Neuronal Activations (NeurIPS 2025 · similarity 0.89)
- Sparsing Law: Towards Large Language Models with Greater Activation Sparsity (ICML 2025 · similarity 0.86)
- LESS: Selecting Influential Data for Targeted Instruction Tuning (ICML 2024 · similarity 0.86)
- Efficient stagewise pretraining via progressive subnetworks (ICLR 2025 · similarity 0.85)
- LensLLM: Unveiling Fine-Tuning Dynamics for LLM Selection (ICML 2025 · similarity 0.85)
- Towards Neuron Attributions in Multi-Modal Large Language Models (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
