Research map / Learning theory and algorithms / Prediction and decision losses

SC$^3$: A Multi-Solvent Solubility Challenge and Benchmark

NeurIPS 2026 · Learning theory and algorithms › Prediction and decision losses · approach: Conformal prediction sets

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Builds SC³, a curated multi-solvent solubility benchmark from BigSolDB v2.1 with recalibrated aleatoric floor (0.106 log S), consensus tiers, leakage-checked splits, PS-RMSE/Z-RMSE metrics, and a 31-model evaluation plus data-scaling, QM transfer, and interpretability analyses.

Problem

Multi-solvent solubility models appear to reach the experimental-noise ceiling, yet are not reliable; published benchmarks differ in curation, use count-weighted RMSE that hides tail-solvent failure, and use a worst-case 0.6–0.8 log S inter-lab figure as the aleatoric limit.

Approach

A reproducible curation pipeline (stereo-preserving canonicalisation, two-stage duplicate-DOI detection, cleaning waterfall) plus a recalibrated expected-disagreement aleatoric floor, nested Gold/Silver/Bronze consensus tiers with per-point sigma, and per-solvent-averaged metrics (PS-RMSE, Z-RMSE) that expose failure hidden by count-weighted RMSE.

Key results

• Recalibrated aleatoric floor of 0.106 log S (95% CI 0.093-0.120), roughly 6x tighter than the conventional 0.6-0.8 figure.
• CatBoost on RDKit reaches 0.593 Bronze PS-RMSE, about 5x the aleatoric floor; no deep or foundation model closes this gap, with ChemFM at 0.714 Bronze.
• QM pretraining on CombiSolv-QM improves performance in 9/9 multi-T cells and gives a 5-20x data-efficiency gain at low fine-tuning fractions.

Surprising findings

• Solute, not solvent, carries roughly 70% of SHAP attribution mass across representations.
• ChemFM has the lowest Eval RMSE (0.437) yet trails the best tree model by about 0.15 log S on Bronze PS-RMSE.

Limitations

• Curation required manual interventions (unit-swap corrections, DOI-level outlier reviews) that are subjective and may not generalise to future releases.
• Crowded adjacent benchmark space makes cross-benchmark comparison difficult.
• Diagnoses a representation bottleneck but does not prescribe an architectural solution.

Experimental setup

Datasets: BigSolDB v2.1 CombiSolv-QM

Benchmarks: SC3 Eval SC3 OOD SC3 Gold SC3 Silver SC3 Bronze

Metrics: RMSE R^2 PS-RMSE Z-RMSE MAE MedAE

Baselines: LightGBM CatBoost XGBoost Random Forest Dissolvr FastSolv Chemprop Solvaformer SolubNet MolMerger Uni-Mol2 SolTranNet

Builds on: BigSolDB CombiSolv-QM FastProp Chemprop Apelblat equation

Closest papers by meaning

• CheMixHub: Datasets and Benchmarks for Chemical Mixture Property Prediction (NeurIPS 2025 · similarity 0.85)
• The Catechol Benchmark: Time-series Solvent Selection Data for Few-shot Machine Learning (NeurIPS 2025 · similarity 0.85)
• $\nabla^2$DFT: A Universal Quantum Chemistry Dataset of Drug-Like Molecules and a Benchmark for Neural Network Potentials (NeurIPS 2024 · similarity 0.83)
• DiSCO: Diffusion Schrödinger Bridge for Molecular Conformer Optimization (AAAI 2024 · similarity 0.83)
• BOOM: Benchmarking Out-Of-distribution Molecular Property Predictions of Machine Learning Models (NeurIPS 2025 · similarity 0.83)
• Chemistry-Inspired Diffusion with Non-Differentiable Guidance (ICLR 2025 · similarity 0.83)