Research map / Label-efficient and robust learning / Recognition and long-tail
MD tree: a model-diagnostic tree grown on loss landscape
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MD tree, a fixed-hierarchy decision tree over loss-landscape metrics (training error, mode connectivity, Hessian trace, CKA similarity) that diagnoses whether a pre-trained CIFAR-10 ResNet fails due to optimizer hyperparameter (batch size) or model size/data, trained few-shot and transferred across datasets and scales.
Problem
Diagnosing the root cause of underperformance of a pre-trained neural network (wrong batch size, insufficient model size, insufficient data) without knowing its training configuration, retraining, or relying on validation data.
Approach
Use loss-landscape metrics (mode connectivity, sharpness, similarity) inspired by statistical physics regimes to build a decision tree with a fixed, prior-informed hierarchy (training error, then connectivity, then sharpness/similarity), whose thresholds are learned from few-shot samples, instead of validation-error features and standard information-gain trees.
Key results
- MD tree reaches about 87.7% diagnosis accuracy with 96 training samples in dataset transfer, outperforming validation-based DT by 14.88%.
- Trained on models with fewer than 0.01M parameters, MD tree achieves 82.56% accuracy on the full test set with models up to 44.66M parameters.
- For Q2 (optimizer vs model size), MD tree reaches 74.11% with 160 models, outperforming the validation-based method by 18.31%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use loss-landscape metrics instead of validation metrics for diagnosis | MD tree on Q1 dataset transfer · Q1 dataset transfer, 96 training samples | diagnosis accuracy: 14.88% | Helped |
| Transfer thresholds learned on models with fewer than 0.01M parameters to larger models | MD tree · Q1 scale transfer | diagnosis accuracy: 82.56% | Helped |
| Use loss-landscape metrics instead of validation metrics for diagnosis | MD tree on Q2 dataset transfer · Q2 dataset transfer | diagnosis accuracy: 18.31% | Helped |
| Use the fixed MD tree hierarchy rather than an unconstrained standard decision tree (ablation) | MD tree · Q1 and Q2 few-shot diagnosis | diagnosis accuracy: 12.71% | Helped |
| Use similarity instead of sharpness at the final tree level (ablation) | MD tree · Model diagnosis | diagnosis accuracy | No clear effect |
| Use MD tree to choose the optimizer-hyperparameter direction rather than random diagnosis | Q1 CIFAR-10 one-step change with optimal step size · CIFAR-10 | accuracy improvement: 5.52% vs 3.50% | Helped |
| Use MD tree to choose the one-step configuration adjustment | Q2 CIFAR-10 one-step change with optimal step size · CIFAR-10 | accuracy improvement: 8.10 $\pm$ 0.22 vs 6.59 $\pm$ 0.00 | Helped |
| Measure sharpness using Hessian trace | MD tree measurement pipeline · ResNet18 CIFAR-10 model configuration | runtime increase: 0.244% | Hurt |
| Measure mode connectivity | MD tree measurement pipeline · ResNet18 CIFAR-10 model configuration | runtime increase: 7.04% | Hurt |
| Train on models with at least 0.10M parameters rather than only smaller models | MD tree scale transfer · Q1 scale transfer | diagnosis accuracy: 82.82% to 91.28% | Helped |
Surprising findings
- Validation-based metrics produce nonlinear, poorly transferable decision boundaries, while loss-landscape features are piecewise linearly separable.
- Medium-sized models suffice for diagnosing large models; extremely large training models are not necessary.
Limitations
- Q2 accuracy is lower (79.56% max) than Q1 (89.85%), possibly due to distribution shift from label noise.
- Tree structure is fixed by hand and does not dynamically terminate branching.
- Evaluated mainly on ResNet models on CIFAR-10 with a single hyperparameter family (batch size).
Experimental setup
Datasets: CIFAR-10
Benchmarks: CIFAR-10 CIFAR-10-C CIFAR-10-LT
Metrics: diagnosis accuracy CIFAR-10 test accuracy improvement after one-step change
Baselines: Validation + DT Hyperparameter + DT Loss landscape + DT Random diagnosis Standard decision tree with loss landscape metrics
Builds on: statistical physics of learning regimes mode connectivity Hessian trace sharpness CKA similarity
Closest papers by meaning
- Taxonomizing local versus global structure in neural network loss landscapes (NeurIPS 2021 · similarity 0.83)
- Two Compacted Models for Efficient Model-Based Diagnosis (AAAI 2022 · similarity 0.83)
- Neural Architecture Search in A Proxy Validation Loss Landscape (ICML 2020 · similarity 0.82)
- PICID: A Modular Evaluation Infrastructure for Reproducible PHM Across Tasks and Domains (NeurIPS 2026 · similarity 0.82)
- Unsupervised Model Tree Heritage Recovery (ICLR 2025 · similarity 0.81)
- Multistage Defer Trees for Hybrid Interpretability: If at First You Can't Succeed, Tree Again (NeurIPS 2026 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.