Research map / Probabilistic and scientific modeling / Molecules and proteins
A Theory of Dynamic Benchmarks
Ali Shirali, Rediet Abebe, Moritz Hardt
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Continue sequential path benchmarking beyond three rounds | Path dynamic benchmark · Realizable classification | majority-vote risk: \frac{\epsilon^2}{8} | No clear effect |
| Run additional rounds of path benchmarking | CIFAR-10 path dynamic benchmark · CIFAR-10 | majority-vote risk | No clear effect |
| Run additional rounds of path benchmarking | SNLI path dynamic benchmark · SNLI | majority-vote risk | No clear effect |
| Have rollout error overlaps resemble the constructed negative example more closely | Path dynamic benchmark rollouts · CIFAR-10 | final risk correlation with z_4: Pearson r=0.46, p < 0.001 | Mixed |
| Have rollout error overlaps resemble the constructed negative example more closely | Path dynamic benchmark rollouts · SNLI | final risk correlation with z_6: Pearson r=0.68, p < 0.001 | Mixed |
| Use exponential-loss dynamic gradient-based updates | Dynamic gradient-based updating · Classification | risk upper bound: R_\gD(h_t) \le \exp(-\frac{(1 - 2\epsilon)^2 t}{2}) | Mixed |
Closest papers by meaning
- Measuring What Matters: Synthetic Benchmarks for Concept Bottleneck Models (NeurIPS 2026 · similarity 0.87)
- How Benchmark Prediction from Fewer Data Misses the Mark (NeurIPS 2025 · similarity 0.86)
- Adaptive Data Optimization: Dynamic Sample Selection with Scaling Laws (ICLR 2025 · similarity 0.86)
- DataDecide: How to Predict Best Pretraining Data with Small Experiments (ICML 2025 · similarity 0.85)
- Benchmark Health Index: A Systematic Framework for Benchmarking the Benchmarks of LLMs (NeurIPS 2026 · similarity 0.85)
- Kinetics: Rethinking Test-Time Scaling Law (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
