Research map / Large language models / Language, safety and interpretability
Learning and Evaluating Contextual Embedding of Source Code
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Pre-train a BERT Large code model and fine-tune it instead of using the strongest task-wise BiLSTM with Word2Vec | CuBERT · Five classification tasks | accuracy: 3.2–14.7% | Helped |
| Fine-tune the pre-trained model instead of training a Transformer from scratch | CuBERT · Five classification tasks | accuracy: 95.213145% vs 78.28434%; 92.46354% vs 76.554555%; 93.35517% vs 87.82762%; 98.08504% vs 91.017634%; 79.12152% vs 49.56463 | Helped |
| Use 2 fine-tuning epochs instead of the best 100-epoch BiLSTM baseline | CuBERT · Variable-Misuse Classification | accuracy: 94.042% vs 80.52728% | Helped |
| Reduce fine-tuning data from 100% to 33% (ablation) | CuBERT, 2 epochs · Variable-Misuse Classification | accuracy: 94.042% -> 91.39687% | Hurt |
| Reduce fine-tuning data from 100% to 33% (ablation) | CuBERT, 2 epochs · Function-Docstring Mismatch | accuracy: 97.20764% -> 96.37577% | Hurt |
| Reduce fine-tuning data from 100% to 33% (ablation) | CuBERT, 2 epochs · Exception Type | accuracy: 61.039084% -> 20.08514% | Hurt |
| Increase example length from 128 to 1024 subword tokens (ablation) | CuBERT · Variable-Misuse Classification | accuracy: 83.96647% -> 95.834655% | Helped |
| Increase example length from 128 to 1024 subword tokens (ablation) | CuBERT · Function-Docstring Mismatch | accuracy: 98.188955% -> 97.899705% | Hurt |
| Increase example length from 128 to 1024 subword tokens (ablation) | CuBERT · Exception Type | accuracy: 62.02593% -> 81.26935% | Helped |
| Replace the LSTM sequence encoder with a CuBERT encoder in the pointer model | Variable-Misuse localization and repair · CuBERT test data | Localization+Repair Accuracy: 56.8859616665433% -> 90.84322% | Helped |
| Use CuBERT's lexical representation instead of the published model's richer syntax, data-flow, and control-flow representations | Variable-Misuse localization and repair · H test data | Localization+Repair Accuracy: 73.8% -> 82.90709% | Helped |
Closest papers by meaning
- CODE REPRESENTATION LEARNING AT SCALE (ICLR 2024 · similarity 0.87)
- ObscuraCoder: Powering Efficient Code LM Pre-Training Via Obfuscation Grounding (ICLR 2025 · similarity 0.87)
- Towards A Generalist Code Embedding Model Based On Massive Data Synthesis (NeurIPS 2025 · similarity 0.87)
- LILO: Learning Interpretable Libraries by Compressing and Documenting Code (ICLR 2024 · similarity 0.86)
- CoRNStack: High-Quality Contrastive Data for Better Code Retrieval and Reranking (ICLR 2025 · similarity 0.86)
- Emergent Representations of Program Semantics in Language Models Trained on Programs (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.