Does it actually help? 37 common changes, by what papers report
For each change, the share of papers at ICML, NeurIPS, ICLR, CVPR, AAAI, ICCV, ECCV, TPAMI, IJCV and JMLR (2016–2026) that report a gain or a loss after making it. From 1,254,369 results read from 124,803 papers.
Adding a component: 82% of papers report a gain. Turning a knob: 65%.
Each paper counts once per change, with the outcome it reports most often.
Knobs: a setting turned up or down
| Change | Papers | Report a gain | Report a loss |
|---|---|---|---|
| More training data | 1,147 | 78% | 6% |
| More layers / deeper model | 4,153 | 69% | 17% |
| Larger beam / more samples (best-of-N) | 314 | 68% | 11% |
| More in-context examples | 287 | 68% | 9% |
| Larger model or backbone | 510 | 67% | 13% |
| Longer context window | 432 | 62% | 25% |
| Larger embedding / hidden size | 1,185 | 61% | 18% |
| More diffusion sampling steps | 440 | 61% | 17% |
| Larger batch size | 774 | 60% | 20% |
| Higher weight decay | 88 | 59% | 26% |
| More attention heads | 106 | 58% | 18% |
| More retrieved passages (RAG) | 51 | 57% | 25% |
| Train longer | 369 | 55% | 21% |
| Larger LoRA rank | 233 | 46% | 20% |
| Larger learning rate | 447 | 44% | 38% |
| Classifier-free guidance scale up | 150 | 35% | 36% |
| Higher sampling temperature | 122 | 34% | 42% |
Components: a part added to a method
| Change | Papers | Report a gain | Report a loss |
|---|---|---|---|
| Add a contrastive loss | 1,265 | 95% | 2% |
| Add attention module | 467 | 90% | 5% |
| Knowledge distillation | 2,026 | 89% | 7% |
| Add a reconstruction / auxiliary loss | 950 | 89% | 5% |
| Test-time augmentation | 80 | 89% | 8% |
| Pretrained initialization | 86 | 86% | 5% |
| Add learning-rate warmup | 345 | 84% | 9% |
| Ensembling | 1,047 | 83% | 9% |
| Add residual connections | 533 | 82% | 9% |
| Add data augmentation | 669 | 81% | 9% |
| Add positional encoding | 629 | 81% | 8% |
| Add EMA of weights | 369 | 80% | 9% |
| Self-consistency / majority vote | 561 | 78% | 10% |
| Add mixup or CutMix | 530 | 76% | 17% |
| Cosine learning-rate schedule | 66 | 73% | 14% |
| Add batch / layer normalization | 427 | 72% | 15% |
| Chain-of-thought prompting | 1,101 | 69% | 18% |
| Add dropout | 783 | 69% | 19% |
| Add label smoothing | 188 | 67% | 20% |
| Add gradient clipping | 88 | 64% | 12% |
Check a change for your own setup
The results were read by AI from each paper's full text and matched by wording. These are what papers chose to report, and published results lean toward gains.
