Research map / Probabilistic and scientific modeling / Molecules and proteins
Towards Inferential Reproducibility of Machine Learning Research
Michael Hagmann, Philipp Meier, Stefan Riezler
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add R3F and select the best run | BART · CNN/DailyMail | ROUGE-1: 44.15 -> 44.72 | Improved |
| Add R3F and select the best run | BART · CNN/DailyMail | ROUGE-2: 21.13 -> 21.17 | Improved |
| Add R3F and select the best run | BART · CNN/DailyMail | ROUGE-L: 40.81 -> 41.40 | Improved |
| Evaluate across the reported R3F meta-parameter configurations instead of selecting the best run (ablation) | BART+R3F versus BART · CNN/DailyMail | ROUGE-1: 44.15 -> 41.06 | Hurt |
| Evaluate across the reported R3F meta-parameter configurations instead of selecting the best run (ablation) | BART+R3F versus BART · CNN/DailyMail | ROUGE-2: 21.30 -> 19.00 | Hurt |
| Evaluate across the reported R3F meta-parameter configurations instead of selecting the best run (ablation) | BART+R3F versus BART · CNN/DailyMail | ROUGE-L: 40.84 -> 36.40 | Hurt |
| Use lambda 0.1 rather than other tested lambda values (ablation) | BART+R3F · CNN/DailyMail | ROUGE-2 | Hurt |
| Add R3F and select the best run | BART · RedditTIFU | ROUGE-2: -0.155 effect size | Improved |
| Account for meta-parameter variation in the RedditTIFU comparison (ablation) | BART+R3F versus BART · RedditTIFU | ROUGE-2 effect size: -0.155 -> -0.0617 | Hurt |
| Condition the comparison on readability and word rarity | BART+R3F versus BART · CNN/DailyMail | ROUGE-2 | Mixed |
| Use lambda 0.1 and examine interactions with data properties (ablation) | BART+R3F · CNN/DailyMail | ROUGE-2 | Hurt |
Closest papers by meaning
- Reproduce, Replicate, Reevaluate. The Long but Safe Way to Extend Machine Learning Methods (AAAI 2024 · similarity 0.91)
- Quantifying Uncertainty in Error Consistency: Towards Reliable Behavioral Comparison of Classifiers (NeurIPS 2025 · similarity 0.86)
- Crossing the Validation crisis: Cross-validation reduces benchmarking variance surprisingly well (NeurIPS 2026 · similarity 0.86)
- Position: Causal Machine Learning Requires Rigorous Synthetic Experiments for Broader Adoption (ICML 2025 · similarity 0.86)
- Task-Agnostic Machine-Learning-Assisted Inference (NeurIPS 2024 · similarity 0.86)
- Is Cross-validation the Gold Standard to Estimate Out-of-sample Model Performance? (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
