Research map / Learning theory and algorithms / Combinatorial and search optimization
Pareto-Optimal Fronts for Benchmarking Symbolic Regression Algorithms
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Exhaustively searches GEP K-expressions over 34 SRBench datasets with eight numerical optimizers to build absolute Pareto-optimal (APO) fronts of R² vs. expression length as fixed reference baselines for SR benchmarking.
Problem
Symbolic regression benchmarks evaluate algorithms only relative to each other (ranked Pareto fronts), giving no indication of attainable limits or efficiency of expression accuracy versus length on black-box datasets.
Approach
Use exhaustive search over fixed-length GEP K-expressions with a fixed primitive set and multiple numerical optimizers to extract absolute Pareto-optimal fronts (best R² per expression length) for SRBench datasets, and propose benchmarking conventions that use actual values instead of ranks and per-dataset analysis.
Key results
- Among the 34 APO fronts, 15 are Type I, 5 Type II, 5 Type III and 9 Type IV, indicating many SRBench methods are far from the attainable front on many datasets.
- Only 1.47% of standard deviations of R² across numerical optimizers exceeded 0.01, and the numerical optimizer choice has no strong effect on R² (mean SD 0.00043); Powell produced the APO-front expression most often (32.1%).
- On average 23.7% of expressions two mutations away from the APO front greedily mutate back to it, ranging from 0% (dataset 659) to 68.8% (dataset 690).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Change the numerical optimization method used to optimize ERCs | Exhaustive symbolic-regression search for APO fronts · SRBench datasets | R² score: mean standard deviation 0.00043; median 3.42e-19; maximum 0.043 | No clear effect |
| Use BFGS instead of other numerical optimization methods | Exhaustive symbolic-regression search for APO fronts · SRBench datasets | frequency of producing an expression on the APO front: BFGS: 9.41%; Powell: 32.1%; CG: 25.0%; trust-constr: 20.0% | No clear effect |
| Compare current SRBench algorithms with the exhaustive APO front | SRBench symbolic-regression algorithms · 34 SRBench datasets | R² score at a given expression length | Mixed |
| Randomly sample expressions from the exhaustive search space of short expressions | Symbolic-regression search · SRBench datasets | share of expressions in the top histogram bin: 85.3% have less than 1%; 66.9% have less than 0.1%; 35.2% have less than 0.01% | Hurt |
| Use actual metric values instead of rankings for Pareto analysis | SR benchmarking · SRBench | reliability of Pareto conclusions | Helped |
| Add per-dataset Pareto analysis to aggregated results | SR benchmarking · SRBench | visibility of per-dataset trends | Helped |
Surprising findings
- Only SBP-GP exceeds the APO front's R² in Figure 1, but only at much higher expression length.
- Ranked-axis Pareto analysis can invert which algorithms appear Pareto-optimal when a new algorithm is added (Rank Inversion Paradox).
Limitations
- APO fronts are only absolute with respect to the fixed primitive set {Add, Sub, Mul, Div, Pow} and local numerical optimizers, not a true global optimum.
- Search limited to head length 3 (34 datasets) and head length 4 (one seed, 30 datasets), so longer expressions are not covered.
- Restricted to small datasets (<1000 points, <10 features).
Experimental setup
Datasets: SRBench black-box regression datasets (34 datasets, <1000 points, <10 features) SRBench black-box regression datasets (30 datasets for head length 4)
Benchmarks: SRBench
Metrics: R-squared score expression length Pareto dominance
Baselines: AFP AFP FE AIFeynman BSR DSR EPLEX FEAT FFX GP-GOMEA ITEA MRGP Operon
Builds on: gene expression programming SRBench DistilSR
Closest papers by meaning
- Ab Initio Nonparametric Variable Selection for Scalable Symbolic Regression with Large $p$ (ICML 2025 · similarity 0.88)
- ParFam -- (Neural Guided) Symbolic Regression via Continuous Global Optimization (ICLR 2025 · similarity 0.86)
- SMILE: Bridging Continuous Optimization and Discrete Symbolic Recovery (NeurIPS 2026 · similarity 0.86)
- Symbolic regression via MDLformer-guided search: from minimizing prediction error to minimizing description length (ICLR 2025 · similarity 0.86)
- Improving Monte Carlo Tree Search for Symbolic Regression (NeurIPS 2025 · similarity 0.86)
- Dimension Reduction for Symbolic Regression (AAAI 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.