Special edition · What didn't work · 621,064 changes papers added and measured
Of the 110,393 papers that report adding something to a model, a prompt or a training run, 72,561 (66%) report at least one addition that did not help.
scime atlas has read the full text of 126,758 papers from ICML, NeurIPS, ICLR, CVPR, AAAI, ICCV and ECCV and the journals TPAMI, IJCV and JMLR (2016–2026), and recorded each change a paper tested and what it measured: 1,273,440 findings. This edition keeps the 621,064 findings where a paper added something (a layer, a loss, more data, a larger learning rate) and asks how often it helped.
Across all of them, papers report that what they added helped in 69% of cases, made no clear difference in 10% and made things worse in 16% (5% mixed). Machine-learning papers report a null or negative result for 31% of their changes, computer-vision papers for 22%.
ICML, NeurIPS, ICLR, JMLR · 53,609 papers · did not help in 31%
CVPR, ICCV, ECCV, TPAMI, IJCV · 39,374 papers · did not help in 22%
AAAI · 17,410 papers · did not help in 23%
Share of added changes reported as no clear effect or worse, by publication year. Differences between years are small and can come from how the papers were read; do not read them as a trend.
AI-assisted. What each paper tried and what it measured was extracted by AI from its full text and not checked by hand; some lines are wrong, so open the paper before you cite it. Published papers report what worked more often than what did not: the real share of ideas that fail is higher than anything counted here.
Describe the change you plan to try in scime atlas: "Already tried" shows what papers measured when they made a similar change, including the ones that did not help.
Some changes are tried in thousands of papers. Ranked by how often the papers that tried them report no clear effect or a loss: higher sampling temperature did not help in 52% of the 122 papers that report it; add a contrastive loss in only 4% of 1,317.
Bars: helped, mixed, no clear effect, worse. Each paper counts once.
Knob = turning a setting up (learning rate, temperature, model size); component = adding a part (a loss, a module, augmentation). Each paper's outcome is the one most of its findings on that change report. Every change has its own page under /changes.
Free for signed-in readers
Three more pages: the research topics where the fewest changes help (and where most do), 30 null and negative results from the most cited papers, and every conference and year side by side. Sign in with a free scime account to keep reading.
Already signed in? Reload this page. Accounts are free; we only use your email to sign you in and, if you agree, to tell you about new editions.
Papers. All 137,217 papers of ICML, NeurIPS, ICLR, CVPR, AAAI, ICCV, ECCV, TPAMI, IJCV and JMLR from 2016 to 2026 in scime atlas; for 126,758 of them an AI model read the full text and listed each change the paper tested, on what, the metric, the reported number and whether it helped (improved, no clear effect, worse or mixed). These were not checked by hand.
What is counted. Only changes that add, increase or use something; removals, reductions and swaps are left out because their effect reads the other way round (a removal that hurts means the part helped), and so are proven bounds. Shares count each paper once, split evenly over its findings. "The usual suspects" uses the 37 change patterns and the one-outcome-per-paper rule of the /changes pages, so the numbers match.
Caveat. Papers choose what to report, and results that did not help are often left out or reported only in ablations. These shares describe what gets reported, not the chance that an idea works for you; the real failure rate is higher. Citation counts are from OpenAlex.
Corrections: contact@latenticlabs.com. Not affiliated with or endorsed by any of the conferences or journals.
← All editions · © 2026 Latentic Labs