Research map / Large language models / AI and society

Common Task Framework For a Critical Evaluation of Scientific Machine Learning Algorithms

NeurIPS 2025 · Large language models › AI and society · approach: Benchmarking and reproducibility practice · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes the ctf4science Common Task Framework with hidden-test-set scoring (12 metrics across forecasting, denoising, limited data, parametric generalization) and benchmarks ~14 scientific ML methods plus baselines on Lorenz and Kuramoto-Sivashinsky, with a planned sea surface temperature competition.

Problem

Scientific machine learning lacks standardized, objective benchmarks with withheld test sets, leading to weak baselines, reporting bias, and irreproducible comparisons across methods for dynamical system modeling.

Approach

A Common Task Framework for dynamical systems that scores methods on a twelve-metric suite (short- and long-time forecasting, noisy reconstruction, limited data, parametric interpolation/extrapolation) using hidden test matrices, with a composite score and radar-plot profiles, plus a planned Kaggle-style competition on a global SST dataset.

Key results

• Reservoir computing achieves the highest composite score on KS (18.88) and parametric generalization score of 99.89 on Lorenz, while LSTM gets the best Lorenz average on E1-E6 (78.07) and E7-E10 (41.33).
• On Lorenz, the best composite average is LSTM (64.54) with Reservoir (54.87) second; the naive average baseline scores -4.73, and zero baseline -39.00.
• Foundation model Sundial achieves the highest average among zero-shot models on Lorenz (45.26), while all zero-shot models score poorly on KS (best TabPFN -2.51 but Sundial -0.64 is near baseline).

Surprising findings

• No single model scores well across all tasks; methods such as DeepONet outperform on tasks outside their design intent.
• Several time-series foundation models (e.g., Moirai, LLMTime) score negative or near the naive baselines on KS, while Sundial is best on Lorenz.

Limitations

• Only two canonical chaotic systems (Lorenz, KS) in the launch; SST competition and control tasks are future work.
• Small training datasets by design; large models are disadvantaged.
• Long-time metrics are simple and transparent but ad hoc; composite averaging can mask task-specific strengths.

Experimental setup

Datasets: ODE_Lorenz PDE_KS SST

Benchmarks: ctf4science Lorenz ctf4science Kuramoto-Sivashinsky

Metrics: RMSE short-time forecast power spectrum long-time forecast histogram distribution error composite average score

Baselines: LSTM ODE-LSTM DeepONet Reservoir computing KAN SINDy PyKoopman FNO PINN Baseline Average Baseline Zeros

Builds on: PDEBench The Well CoDBench Ray Tune ASHA

Closest papers by meaning

• OceanCBM: A Concept Bottleneck Model for Mechanistic Interpretability in Ocean Forecasting (NeurIPS 2026 · similarity 0.85)
• Physics-Guided Foundation Model for Scientific Discovery: An Application to Aquatic Science (AAAI 2025 · similarity 0.85)
• Njord: A Probabilistic Graph Neural Network for Ensemble Ocean Forecasting (NeurIPS 2026 · similarity 0.84)
• SeafloorAI: A Large-scale Vision-Language Dataset for Seafloor Geological Survey (NeurIPS 2024 · similarity 0.84)
• Prediction-Augmented Trees for Reliable Statistical Inference (NeurIPS 2026 · similarity 0.84)
• FACTS: A Factored State-Space Framework for World Modelling (ICLR 2025 · similarity 0.84)