Research map / Architectures and efficiency / RNNs and normalization
Investigating Data Pruning for Pretraining Biological Foundation Models at Scale
Yifan Wu, Jiyue Jiang, Xichen Ye, Yiqi Wang, Chang Zhou, Yitao Xu, Jiayang Chen, He Hu, Weizhong Zhang, Cheng Jin, Jiao Yuan, Yu Li
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use CCI coreset (0.2M) instead of random 0.2M | RNA-FM pretraining · CRISPR On-Target | Spearman corr (%): 26.67 -> 32.90 | Improved |
| use CCI coreset instead of random 0.2M | RNA-FM pretraining · RNA Type Classification | accuracy (%): 82.15 -> 82.88 | Improved |
| use Top I coreset instead of random 0.2M | RNA-FM pretraining · RNA distance map | R2 (%): 76.71 -> 79.25 | Improved |
| use Top I coreset instead of random 0.2M | RNA-FM pretraining · RNA contact map (L:1.0L) | top precision (%): 94.18 -> 94.36 | Improved |
| use influence coresets of 0.2M instead of random 2M (10x more data) | RNA-FM pretraining · RNA Type Classification | accuracy (%): 82.21 -> 82.88 | Improved |
| add fine-tuning on random subset before influence estimation (CCI) (ablation) | influence-guided selection · CRISPR On-Target | Spearman corr (%): 26.96 -> 32.90 | Improved |
| add fine-tuning before influence estimation (Top I) (ablation) | influence-guided selection · RNA Type Classification | accuracy (%): 81.07 -> 82.51 | Improved |
| use CCI coreset instead of random 0.2M | RNA-FM pretraining · RBP-RNA interaction | AUC (%): 75.97 -> 76.04 | No clear effect |
| use Top I vs CCI coreset | RNA-FM pretraining · bpRNA secondary structure | F1 (%): 57.05 vs 56.36 | No clear effect |
| use CCI coreset instead of random 0.2M | ESM-C pretraining · Binary localization | accuracy (%): 73.64 -> 79.25 | Improved |
| use CCI coreset instead of random 0.2M | ESM-C pretraining · Secondary structure (protein) | accuracy (%): 66.18 -> 71.48 | Improved |
| use influence coreset 0.2M vs full pretraining data | ESM-C pretraining · Binary localization | accuracy (%): 91.63 -> 79.25 | Hurt |
Closest papers by meaning
- Beyond Efficiency: Molecular Data Pruning for Enhanced Generalization (NeurIPS 2024 · similarity 0.88)
- What DNA Foundation Models Learn Beyond Sequence Composition (NeurIPS 2026 · similarity 0.86)
- Pruning neural network models for gene regulatory dynamics using data and domain knowledge (NeurIPS 2024 · similarity 0.86)
- Training Compute-Optimal Protein Language Models (NeurIPS 2024 · similarity 0.85)
- Specialized Foundation Models Struggle to Beat Supervised Baselines (ICLR 2025 · similarity 0.85)
- Context-Aware Regularization with Markovian Integration for Attention-Based Nucleotide Analysis (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
