Research map / Label-efficient and robust learning / Recognition and long-tail
How NOT to benchmark your SITE metric: Beyond Static Leaderboards and Towards Realistic Evaluation.
Prabhant Singh, Sibylle Hess, Joaquin Vanschoren
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace LogME with a fixed dataset-agnostic static ranking | LogME · Standard SITE benchmark | weighted Kendall tau: 0.57 -> 0.91 | Improved |
| remove largest models (ResNet-152, ResNet-101, DenseNet-169, DenseNet-201), keeping 7 of 11 models (ablation) | SITE metrics · Standard SITE benchmark | weighted Kendall tau: below 0.6 | Hurt |
| replace average LogME with static ranking on Aircraft | LogME · Aircraft | weighted Kendall tau: 0.41 -> 0.84 | Improved |
| replace NLEEP with static ranking on Aircraft | NLEEP · Aircraft | weighted Kendall tau: -0.51 -> 0.84 | Improved |
| replace H-Score with static ranking on DTD | H-Score · DTD | weighted Kendall tau: 0.04 -> 0.99 | Improved |
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.
- LogME — LogME: Practical Assessment of Pre-trained Models for Transfer Learning (ICML 2021) · better on 7 of 7 shared results
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- How Stable Are Transferability Metrics Evaluations? (ECCV 2022 · similarity 0.88)
- A Theory of Dynamic Benchmarks (ICLR 2023 · similarity 0.86)
- How Benchmark Prediction from Fewer Data Misses the Mark (NeurIPS 2025 · similarity 0.84)
- BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practices (NeurIPS 2024 · similarity 0.84)
- Benchmark Health Index: A Systematic Framework for Benchmarking the Benchmarks of LLMs (NeurIPS 2026 · similarity 0.84)
- Measuring What Matters: Synthetic Benchmarks for Concept Bottleneck Models (NeurIPS 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
