AI Paper Atlas

Research map / Large language models / AI and society

Improving Reproducibility in Machine Learning Research(A Report from the NeurIPS 2019 Reproducibility Program)

Joelle Pineau, Philippe Vincent-Lamarre, Koustuv Sinha, Vincent Lariviere, Alina Beygelzimer, Florence d'Alche-Buc, Emily Fox, Hugo Larochelle

JMLR 2021 · Large language models › AI and society · approach: Benchmarking and reproducibility practice · 317 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
expect (not require) code submission for accepted papers, versus earlier supplementary-only policyNeurIPS conference policy · Code at camera-ready% papers with code: <50% -> 74.4%Improved
voluntary code policy on submission volumeNeurIPS 2019 · Submissionspapers submitted: 4856 -> 6743No clear effect
code available at submission (per checklist)NeurIPS 2019 reviews · Reviewer scorereviewer score: positively associated, p<1e-08Improved
authors from industry versus academia under code policyNeurIPS 2019 acceptance · Acceptance rateacceptance rateNo clear effect
reviewers having code available during reviewNeurIPS 2019 reviewers · Reviewer surveyreviewers finding code useful: 5298 provided, 2255 looked, 1315 usefulMixed
reviewers whose submissions lacked codeNeurIPS 2019 reviewers · Reviewer surveywished code available: 3881Improved
reviewers finding checklist usefulNeurIPS 2019 reviews · Reviewer score and acceptancepaper score, acceptance: 34% found useful; higher scores and acceptanceImproved
checklist answer yes versus noNeurIPS 2019 acceptance · Acceptance rate per questionacceptance rateImproved
checklist NA answers on figures and tablesNeurIPS 2019 acceptance · Acceptance rateacceptance rateNo clear effect
shift from ICLR 2019 to NeurIPS 2019 challengeReproducibility Challenge · Participationpapers claimed: 90 -> 173Improved
challenge year ICLR 2019 to NeurIPS 2019Reproducibility Challenge · Participationparticipating institutions: 35 -> 73Improved
ICML 2019 soft encouragement versus NeurIPS 2019 expectationcode at camera-ready · Code submission% papers with code: 67% -> 74.4%Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy