Research map / Architectures and efficiency / Pruning and efficient tuning
Towards Foundation Models for Scientific Machine Learning: Characterizing Scaling and Transfer Behavior
Shashank Subramanian, Peter Harrington, Kurt Keutzer, Wahid Bhimji, Dmitriy Morozov, Michael W. Mahoney, Amir Gholami
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| pre-train on SYS-1(1,5) then fine-tune vs train from scratch | FNO · SYS-1(5,10) downstream | examples to reach error 1e-2: 8K -> 64 | Improved |
| pre-train and fine-tune vs scratch with large downstream data (~2^15 examples) | FNO · SYS-1/SYS-2 downstream | test error | No clear effect |
| scale model from 64K to 256M parameters with fine-tuning | FNO · SYS-1(5,10), SYS-2(1,2) | test error | Improved |
| pre-train on a 64K-parameter model | FNO · SYS-1(5,10) | test error | Hurt |
| increase distribution shift of downstream physics to large OOD | FNO pre-trained · SYS-1(10,20) | test error | Mixed |
| pre-train on mixture of Poisson, Advection-Diffusion and Helmholtz | FNO · SYS-1/2/3 downstream | test error | Improved |
| remove per-instance input normalization (ablation) | FNO · SYS-2 | performance | Improved |
Closest papers by meaning
- Specialized Foundation Models Struggle to Beat Supervised Baselines (ICLR 2025 · similarity 0.88)
- A Hitchhiker's Guide to Scaling Law Estimation (ICML 2025 · similarity 0.87)
- Scaling Laws for Optimal Data Mixtures (NeurIPS 2025 · similarity 0.87)
- Prescriptive Scaling Reveals the Evolution of Language Model Capabilities (ICML 2026 · similarity 0.87)
- An Empirical Study on Noisy Data and LLM Pretraining Loss Divergence (NeurIPS 2026 · similarity 0.87)
- Quick-Tune: Quickly Learning Which Pretrained Model to Finetune and How (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
