AI Paper Atlas

Research map / Large language models / AI and society

CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Tasks

Ruchir Puri, David S Kung, Geert Janssen, Wei Zhang, Giacomo Domeniconi, Vladimir Zolotov, Julian Dolby, Jie Chen, Mihir Choudhury, Lindsey Decker, Veronika Thost, Luca Buratti and 5 more

NeurIPS 2021 · Large language models › AI and society · approach: Benchmarking and reproducibility practice · 72 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Retain token order with a CNN instead of using a bag of tokensCode classification · C++1400accuracy: 64.50±0.13 -> 93.71±0.18Improved
Add a virtual node to the SPT graph (ablation)GCN code classifier · C++1400accuracy: 95.26±0.13 -> 95.73±0.07Improved
Add a virtual node to the SPT graph (ablation)GIN code classifier · C++1400accuracy: 95.95±0.13 -> 96.36±0.10Improved
Increase GNN depth beyond 5 layers (ablation)SPT code classification · CodeNet benchmarksaccuracyNo clear effect
Use code graph representation instead of SPT (ablation)GCN code classifier · Java250accuracy: 92.70±0.25 -> 94.10±.001Improved
Use code graph representation instead of SPT (ablation)GCN code classifier · Python800accuracy: 93.82±0.16 -> 87.80±.007Hurt
Retain token order and use a Siamese network instead of bag-of-tokens MLPCode similarity · C++1400accuracy: 86.54±0.07 -> 96.56±0.07Improved
Use GMN cross-graph attention instead of MISIM structure-only GNN (ablation)Java250 similarity · Java250 (p4, s5)MAP@R: 0.472±0.023 -> 0.679±0.056Improved
Add MISIM node attributes to GMN (ablation)Java250 similarity GMN · Java250 (p4, s5)MAP@R: 0.679±0.056 -> 0.985±0.015Improved
Train MISIM on cleaned and de-duplicated C++1000 instead of GCJ-297MISIM · POJ-104 test setMAP@R: 12% higherImproved
Train MISIM on C++1000 instead of GCJ-297MISIM · GCJ-297 validation setMAP@R: 10% higher validation score with GCJ-297Hurt
Remove one of two similar problems from C++ classification data (ablation)C-BERT · C++ classificationaccuracyImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy