Research map / Large language models / AI and society
CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| code pretraining (CodeBERT vs RoBERTa) | BERT-style encoder · Clone detection overall | F1/MAP average: 87.4 -> 90.4 | Helped |
| code pretraining (CodeBERT vs RoBERTa) | BERT-style encoder · Devign | accuracy: 61.05 -> 62.08 | Helped |
| code pretraining (CodeBERT vs RoBERTa) | BERT-style encoder · Cloze test | accuracy: 62.45 -> 85.66 | Helped |
| initialize CodeGPT from GPT-2 (domain-adaptive) instead of from scratch (ablation) | CodeGPT · Code completion | overall score: 70.65 -> 71.28 | Helped |
| initialize CodeGPT from GPT-2 (domain-adaptive) instead of from scratch (ablation) | CodeGPT · CONCODE | CodeBLEU: 32.71 -> 35.98 | Helped |
| code pretraining (CodeBERT vs RoBERTa) | bi-encoder code search · CodeSearchNet AdvTest | MRR: 18.33 -> 27.19 | Helped |
| initialize encoder with CodeBERT vs RoBERTa (code) | Encoder-Decoder · CodeTrans C#->Java | CodeBLEU: 80.18 -> 79.41 | Mixed |
| initialize encoder with CodeBERT instead of random | Encoder-Decoder · Bugs2Fix | exact match accuracy: 0.092 -> 0.108 | Helped |
| code pretraining (CodeBERT vs RoBERTa encoder init) | Encoder-Decoder · Code summarization | smoothed BLEU: 16.57 -> 17.83 | Helped |
| initialize encoder with XLM-R | multilingual Transformer · Microsoft Docs | BLEU: 52.67 -> 66.16 | Helped |
| normalize function and variable names in test queries | CodeBERT · CodeSearchNet Python | MRR: 0.869 -> 0.507 | Hurt |
Closest papers by meaning
- CodeMMLU: A Multi-Task Benchmark for Assessing Code Understanding & Reasoning Capabilities of CodeLLMs (ICLR 2025 · similarity 0.89)
- DyCodeEval: Dynamic Benchmarking of Reasoning Capabilities in Code Large Language Models Under Data Contamination (ICML 2025 · similarity 0.88)
- Teaching Large Language Models to Self-Debug (ICLR 2024 · similarity 0.87)
- BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions (ICLR 2025 · similarity 0.87)
- LLM-Assisted Code Cleaning For Training Accurate Code Generators (ICLR 2024 · similarity 0.86)
- Unsupervised Evaluation of Code LLMs with Round-Trip Correctness (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.