Research map / Large language models / AI and society
Reproduce, Replicate, Reevaluate. The Long but Safe Way to Extend Machine Learning Methods
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Reproduces, replicates (PyTorch reimplementation) and reevaluates (Cora, Pubmed) the KENN knowledge-enhanced neural network experiments on Citeseer before extending it to graphs, distilling nine lessons learned.
Problem
Reimplementing and extending a machine learning method (KENN, from TensorFlow to PyTorch Geometric) is risky because reported results may not be reproducible, hyperparameters are underspecified, and results may not generalize across datasets.
Approach
A progressive reproduce → replicate → reevaluate workflow, with explicit qualitative and quantitative reproducibility targets, applied before extending a method; the paper documents the pitfalls encountered at each step and derives lessons learned.
Key results
- On Citeseer, KENN significantly outperformed the base neural network at all training sizes (p << 0.01) in the reproduction, consistent with the original claim H1.
- The replication in PyTorch confirmed H1 but did not replicate the trend H2 that knowledge-enhancement gain grows as training data shrinks.
- On Cora, KENN improved over the base network (H1 supported), but on Pubmed no significant gain was found and quantitative reproducibility was achieved only for Pubmed between implementations.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add KENN knowledge-enhancement layers to the base neural network | NN on Citeseer · Citeseer, 10%, 25%, 50%, 75%, and 90% train | accuracy: 0.108; 0.073; 0.065; 0.055; 0.049 | Helped |
| Add KENN knowledge-enhancement layers to the base neural network | NN replication on Citeseer · Citeseer, 10%, 25%, 50%, 75%, and 90% train | accuracy: 0.079; 0.044; 0.060; 0.056; 0.058 | Helped |
| Add KENN knowledge-enhancement layers to the base neural network | NN on Cora · Cora, 10%, 25%, 50%, 75%, and 90% train | accuracy: 0.220; 0.193; 0.187; 0.159; 0.156 | Helped |
| Add KENN knowledge-enhancement layers to the base neural network | NN on Pubmed · Pubmed, 10%, 25%, 50%, 75%, and 90% train | accuracy: 0.071; 0.075; 0.033; 0.051; 0.004 | Mixed |
| Use the same KENN hyperparameters on additional datasets rather than tuning them per dataset | KENN · Cora and Pubmed | accuracy | Mixed |
Surprising findings
- Reexecuting the initial implementation with seeds still yielded non-identical numbers, due to a non-deterministic set operation and framework differences.
- The default Glorot-uniform vs. PyTorch uniform linear-layer initialization differed between frameworks, a detail found only by isolating components.
Limitations
- Only the transductive part of the KENN experiments was studied; baselines RNM and SBR were not reproduced.
- KS-test can only provide evidence of inequality, and no community standard for quantitative reproducibility was used.
- Hyperparameters were not tuned per dataset, so KENN results on Cora and Pubmed are not competitive with SOTA.
Experimental setup
Datasets: Citeseer Cora Pubmed
Benchmarks: Citeseer Cora Pubmed
Metrics: test accuracy mean difference between KENN and base NN one-sided t-test p-value Kolmogorov-Smirnov test p-value
Baselines: KENN Relational Neural Machines Semantic-Based Regularization
Builds on: KENN Relational Neural Machines Semantic-Based Regularization
Closest papers by meaning
- The Emergence of Reproducibility and Consistency in Diffusion Models (ICML 2024 · similarity 0.86)
- Benchmark Data Repositories for Better Benchmarking (NeurIPS 2024 · similarity 0.86)
- Position: Let’s Strengthen Verifiability if We Can’t Enforce Reproducibility (NeurIPS 2026 · similarity 0.86)
- Position: Causal Machine Learning Requires Rigorous Synthetic Experiments for Broader Adoption (ICML 2025 · similarity 0.86)
- When to retrain a machine learning model (ICML 2025 · similarity 0.86)
- What Do Machine Learning Researchers Mean by “Reproducible”? (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.