Research map / Learning theory and algorithms / Prediction and decision losses

What Do Machine Learning Researchers Mean by “Reproducible”?

AAAI 2025 · Learning theory and algorithms › Prediction and decision losses · approach: Data attribution and recourse · 4 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Survey-based position paper that clarifies the many meanings of "reproducibility" in AI/ML by proposing eight distinct scientific-rigor topics and mapping their relationships, based on a review of 101 papers.

Problem

The term "reproducibility" is used inconsistently in AI/ML research, making it unclear which scientific rigor issues the community is addressing.

Approach

Refines the ACM repeatability/reproducibility/replicability terminology into eight rigor topics (adding adaptability, model selection, label/data quality, meta & incentives, and maintainability), grounded in a categorization of self-identified reproducibility papers and a historical literature review, and maps how these topics interact.

Key results

• Review of 101 papers since 2017 yields eight rigor topics, with model selection (19.8% of papers) and reproducibility (16.8%) as the largest categories.
• Adaptability and label/data quality each account for only 4.0% of papers, indicating they are understudied.
• Proposes direct relationships (repeatability to reproducibility to replicability; repeatability and maintainability) and indirect influences of model selection and label quality on the other topics.

Surprising findings

• Adaptability is studied very little (4.0% of papers) despite being a major practical concern, which the authors call a glaring shortfall in the literature.
• Empirical replicability is described as highly understudied, with the authors aware of little AI/ML work beyond a few reimplementation studies.

Limitations

• Corpus limited to 101 papers that self-identify as reproducibility-themed, excluding many works on the same themes.
• Category assignments are the authors' subjective judgment of each paper's most prominent theme.

Experimental setup

Datasets: 101 papers self-identified as about repeatability, reproducibility, or replicability

Metrics: percentage of reviewed papers per topic

Builds on: ACM repeatability/reproducibility/replicability terminology

Closest papers by meaning

• Reproduce, Replicate, Reevaluate. The Long but Safe Way to Extend Machine Learning Methods (AAAI 2024 · similarity 0.85)
• Position: Let’s Strengthen Verifiability if We Can’t Enforce Reproducibility (NeurIPS 2026 · similarity 0.85)
• Position: Machine Learning Conferences Should Establish a "Refutations and Critiques" Track (NeurIPS 2025 · similarity 0.82)
• The Unreasonable Effectiveness of Open Science in AI: A Replication Study (AAAI 2025 · similarity 0.82)
• Position: Rethinking Explainable Machine Learning as Applied Statistics (ICML 2025 · similarity 0.80)
• Towards Trustworthy, Efficient, and Scalable Machine Learning (AAAI 2025 · similarity 0.79)