Research map / Label-efficient and robust learning / Self-supervised and contrastive
CODE REPRESENTATION LEARNING AT SCALE
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CodeSage, a two-stage pretraining scheme for code encoders (DOBF plus full-mask MLM, then bimodal contrastive learning with hard negatives and hard positives) at 130M, 356M, and 1.3B parameters, outperforming prior code encoders on zero-shot code search and classification.
Problem
Existing code representation (embedding) encoders are small (around 125M parameters) and trained on limited corpora, and standard BERT-style MLM masking (80-10-10) is suboptimal for source code.
Approach
Combine identifier deobfuscation with full-mask MLM (no 80-10-10 corruption) in stage I, then contrastive learning on (docstring summary, function body) pairs with distance-based hard negative weighting and hard positives formed by removing the function signature and return statements.
Key results
- CodeSage-small attains 23.19% relative (4.91% absolute) average Code2Code in-language MAP gain over UnixCoder; CodeSage-large reaches 38.51 average MAP vs 27.33 for OpenAI-Ada-002.
- On NL2Code, CodeSage-small improves CoSQA MRR to 49.92 vs 44.23 for OpenAI-Ada-002, and CodeSage-large reaches 52.67 on AdvTest vs 38.08.
- Full-mask MLM (no 80-10-10) consistently outperforms the 80-10-10 corruption on downstream tasks, and the 15% masking rate is best among variants tested.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Full Mask instead of 80-10-10 masking (ablation) | CodeSage pretraining · downstream tasks | performance | Helped |
| combine DOBF and random masking in parallel rather than sequentially (ablation) | CodeSage-base · Code2Code (In) | MAP (%): 25.3 -> 29.2 | Helped |
| add DOBF to random masking (ablation) | CodeSage pretraining · NL2Code search and classification | performance | Helped |
| add random masking to DOBF (ablation) | CodeSage pretraining · Code2Code in-language and cross-language search | performance | Helped |
| add hard negatives (ablation) | bimodal contrastive learning · semantic search | performance | Helped |
| add hard positives by removing function signatures and return statements (ablation) | bimodal contrastive learning · semantic search | performance | Helped |
| add hard positives and hard negatives together (ablation) | bimodal contrastive learning · semantic search | performance | Helped |
| use dropout-based unimodal rather than bimodal contrastive learning (ablation) | CodeSage pretraining · semantic search with long training | performance | Hurt |
| add Stage-II contrastive learning after Stage-I token-level denoising (ablation) | CodeSage · cross-lingual semantic search | performance | Helped |
| train with contrastive learning only from scratch, without token-level denoising (ablation) | CodeSage · multilingual in-language code search | performance scaling with model size | Hurt |
| use 15% masking rather than 7.5%, 30%, or dynamic [10%, 50%] masking (ablation) | CodeSage-base with Full Mask · zero-shot Code2Code search | performance | Helped |
Surprising findings
- Standard BERT 80-10-10 masking hurts code representation learning, with the degradation more severe for code than for text.
- Contrastive learning from scratch without the token-level denoising stage does not scale with model size and training loss converges at a large value.
Limitations
- CodeSage underperforms OpenAI-Ada-002 and UnixCoder on code defect detection under frozen-encoder linear probing.
- Decoder-only baselines perform poorly on semantic search and were not fine-tuned, so comparisons are limited.
- Model size of OpenAI-Ada-002 is unknown, complicating size-matched comparisons.
Experimental setup
Datasets: The Stack CodeSearchNet CodeNet CoSQA AdvTest CodeXGLUE Defect Code Complexity Python Runtime Errors
Benchmarks: Code2Code search (CodeNet-based, nine languages) CoSQA AdvTest CSN Code Defect Code Complexity RunTime error prediction
Metrics: MAP MRR F1 macro
Baselines: CodeBERT GraphCodeBERT StarEncoder UnixCoder OpenAI-Embedding-Ada-002 CodeGen2.5 (7B) Starcoder (15.5B) CodeT5+ (16B) Encoder
Builds on: BERT SimCLR DOBF SimCSE UnixCoder
Closest papers by meaning
- Towards A Generalist Code Embedding Model Based On Massive Data Synthesis (NeurIPS 2025 · similarity 0.89)
- Uncertainty-Aware Contrastive Learning with Hard Negative Sampling for Code Search Tasks (AAAI 2025 · similarity 0.89)
- CoRNStack: High-Quality Contrastive Data for Better Code Retrieval and Reranking (ICLR 2025 · similarity 0.87)
- NextCoder: Robust Adaptation of Code LMs to Diverse Code Edits (ICML 2025 · similarity 0.87)
- ObscuraCoder: Powering Efficient Code LM Pre-Training Via Obfuscation Grounding (ICLR 2025 · similarity 0.87)
- Uncovering LLM-Generated Code: A Zero-Shot Synthetic Code Detector via Code Rewriting (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.