Research map / Large language models / Language, safety and interpretability
LCDB 1.1: A Database Illustrating Learning Curves Are More Ill-Behaved Than Previously Thought
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds LCDB 1.1, a higher-resolution, leakage-controlled sample-wise learning curve database with 265 OpenML datasets and 32 learners, and uses significance-tested detectors to show about 15% of curves are ill-behaved (non-monotone or non-convex).
Problem
Learning curves are assumed to be monotone and convex, but prior large-scale evidence (LCDB 1.0) was low-resolution, had missing data and data leakage, and used non-robust violation tests, so the prevalence of ill-behavior is unclear.
Approach
Construct a denser (4x anchors), more curated database with modern tabular learners, feature-scaling variants and leakage/no-leakage versions, and detect monotonicity/convexity/peaking/dipping violations with all-pairs Bonferroni-corrected paired t-tests that account for log-scale anchors.
Key results
- About 14.9% of LCDB 1.1 CC-18 learning curves (no feature scaling) are significantly ill-behaved, almost twice the rate implied by prior estimates, with 9.9% non-monotone and 11.5% non-convex.
- TabNet is ill-behaved in 74.3% of FULL curves, Sigmoid SVM in 58.1%, while CatBoost is ill-behaved in only 1.5%.
- Feature scaling rarely removes ill-behavior; the mean log MSE of curve fits is over ten times larger for non-monotone and non-convex curves than for well-behaved ones.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase learning-curve anchor resolution from LCDB 1.0 to LCDB 1.1 | Learning curve shape detection · LCDB 1.1 CC-18 versus LCDB 1.0 | ill-behaved curves: 8.1% -> 14.9% | Mixed |
| use min-max feature scaling (ablation) | LCDB 1.1 CC-18 · LCDB 1.1 CC-18 | ill-behaved curves: 14.9% -> 13.5% | Helped |
| use standardization (ablation) | LCDB 1.1 CC-18 · LCDB 1.1 CC-18 | ill-behaved curves: 14.9% -> 11.2% | Helped |
| use feature scaling (ablation) | Most learners · LCDB 1.1 | ill-behavior | No clear effect |
| use min-max feature scaling (ablation) | Ridge Classifier · LCDB 1.1 FULL | ill-behaved curves: 17.0 -> 4.2 | Helped |
| use standardization (ablation) | MLP · LCDB 1.1 FULL | ill-behaved curves: 27.9 -> 16.6 | Helped |
| use min-max feature scaling (ablation) | Nearest Centroid · LCDB 1.1 FULL | ill-behaved curves: 8.3 -> 21.9 | Hurt |
| use standardization (ablation) | LDA · LCDB 1.1 | learning-curve shape | No clear effect |
| fit a parametric curve on non-monotone or non-convex curves rather than well-behaved curves | POW4 curve fitting · LCDB 1.1 | mean squared error | Hurt |
| select among learners with frequently crossing curves rather than rarely crossing curves | Successive Halving · LCDB 1.1 | model-selection performance | Hurt |
| use standardization (ablation) | MLP · LCDB 1.1 CC-18 | phase-transition behavior | Helped |
| use feature scaling (ablation) | SVMs · LCDB 1.1 | error rate | Helped |
Surprising findings
- Naive Bayes, KNN and Nearest Centroid show ill-behavior that was not previously expected in realistic settings.
- Ensemble methods are generally well-behaved, yet severe dipping appears on one OpenML dataset (41027).
Limitations
- Analysis is limited to error rate learning curves for classification with default hyperparameters; hyperparameter tuning effects are not studied.
- Bonferroni correction is pessimistic and inconsistent across ill-behavior types; Holm's method raises the ill-behaved fraction to 19%.
- Some QDA and LDA curves are non-reproducible due to numerical non-determinism; TabPFN v2 covers only a subset of datasets.
Experimental setup
Datasets: OpenML CC-18 LCDB 1.0 datasets
Benchmarks: LCDB 1.1 CC-18 LCDB 1.1 FULL LCDB 1.0
Metrics: error rate F1 AUC log-loss MSE of curve fitting regret monotonicity violation rate convexity violation rate peaking rate dipping rate
Baselines: LCDB 1.0 Dummy classifier CatBoost RealMLP TabNet TabPFN v2
Builds on: LCDB 1.0 OpenML CC-18 Successive Halving TabPFN v2 CatBoost
Closest papers by meaning
- A Hitchhiker's Guide to Scaling Law Estimation (ICML 2025 · similarity 0.85)
- From Kernels to Features: A Multi-Scale Adaptive Theory of Feature Learning (ICML 2025 · similarity 0.84)
- In-Context Learning Strategies Emerge Rationally (NeurIPS 2025 · similarity 0.84)
- Scaling Reward Modeling without Human Supervision (NeurIPS 2026 · similarity 0.84)
- The Impact of Coreset Selection on Spurious Correlations and Group Robustness (NeurIPS 2025 · similarity 0.84)
- Learning curves theory for hierarchically compositional data with power-law distributed features (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.