Research map / Label-efficient and robust learning / Recognition and long-tail
Towards A Generalist Code Embedding Model Based On Massive Data Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces CodeR, a 1.5B code embedding model trained on the synthetic CodeR-Pile dataset (2.9M samples, 47 tasks) with a three-stage weak-to-strong curriculum called Annealing, reaching state-of-the-art on CoIR and CodeRAG.
Problem
Code embedding models for retrieval-augmented code tasks are limited by scarce high-quality training data, since existing embedding training data is text-centric and does not capture code-specific semantic relationships.
Approach
A DRU-guided synthetic data pipeline (LLM brainstorming of 47 code retrieval tasks, lightweight LLM triplet generation with real-corpus hard-negative mining) combined with a three-stage Annealing curriculum that moves from text-only to mixed to code-only data.
Key results
- CodeR-1.5B (text-only + full code data) reaches 81.77 average NDCG@10 on CoIR, above Voyage-code-003 (78.53).
- On CodeRAG, CodeR-1.5B with full code data averages 72.8, the best reported, versus 67.0 for SFR-mistral-7B.
- CodeR trained only on synthetic data reaches 70.12 on CoIR, 2.7 points above CodeXEmbed-2B (general) (67.41).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train with full code data, integrating synthetic and existing retrieval data | CodeR-1.5B · CoIR | NDCG@10 (Avg): 79.77 -> 81.77 | Helped |
| train with full code data, integrating synthetic and existing retrieval data | CodeR-1.5B · CoIR-filter | NDCG@10 (Avg): 91.08 -> 93.00 | Helped |
| train with full code data, integrating synthetic and existing retrieval data | CodeR-1.5B · CodeRAG | NDCG@10 (Avg): 70.9 -> 72.8 | Helped |
| add code retrieval training data to text-only data | CodeR-1.5B · CoIR | NDCG@10 (Avg): 70.12 -> 81.77 | Helped |
| use all four major task types rather than only Text-to-Code Retrieval (ablation) | CodeR-1.5B · CoIR | NDCG@10 (Avg): 65.65 -> 67.64 | Helped |
| use GPT-4o mini rather than Qwen2.5-Coder-32B-Instruct to generate training samples (ablation) | CodeR-1.5B · CoIR (7 tasks) | NDCG@10 (Avg): 67.29 -> 68.95 | Helped |
| replace mined hard negatives with LLM-generated hard negatives (ablation) | CodeR-1.5B · CoIR (7 tasks) | NDCG@10 (Avg): 67.29 -> 56.40 | Hurt |
| replace mined hard negatives with 1 LLM-generated hard negative (ablation) | CodeR-1.5B · CoIR (7 tasks) | NDCG@10 (Avg): 67.29 -> 59.38 | Hurt |
| add the second-stage mixed-data training after Stage-1 (ablation) | CodeR-1.5B · CoIR | NDCG@10 (Avg): 63.14 -> 69.13 | Helped |
| add the final code-only Stage-3 training (ablation) | CodeR-1.5B · CoIR | NDCG@10 (Avg): 69.13 -> 70.12 | Helped |
| add the three-stage Annealing curriculum instead of direct mixed-data training (ablation) | CodeR-1.5B · CoIR | NDCG@10 (Avg): 68.49 -> 70.12 | Helped |
| add text-only training data (ablation) | CodeR-1.5B · CoIR | NDCG@10 (Avg): 68.89 -> 70.12 | Helped |
Surprising findings
- GPT-4o mini-generated training samples yield only a 1.66-point gain over Qwen2.5-Coder-32B-Instruct-generated ones, suggesting lightweight open LLMs suffice for generation.
- Qwen2.5-Coder-32B-Instruct annotation accuracy on label-0 pairs is only 38%, lower than GPT-4o mini's 73%.
Limitations
- Supports only Chinese and English natural languages.
- Only a 1.5B-scale model is released; other scales not explored.
- Synthetic data could be extended to re-ranking models but this is not done.
Experimental setup
Datasets: CodeR-Pile BGE training set CoIR training datasets GitHub-code-clean
Benchmarks: CoIR CoIR-filter CodeRAG
Metrics: NDCG@10
Baselines: BGE-M3 E5-Mistral-7B-Instruct Gemini-embedding UniXcoder Jina-v2-code CodeXEmbed Voyage-code-002 Voyage-code-003 SFR-mistral-7B BGE-base
Builds on: E5-Mistral CoIR Qwen2.5-Coder Topk-PercPos InfoNCE
Closest papers by meaning
- CodeMMR: Bridging Natural Language, Code, and Image for Unified Retrieval (CVPR 2026 · similarity 0.90)
- CODE REPRESENTATION LEARNING AT SCALE (ICLR 2024 · similarity 0.89)
- CoRNStack: High-Quality Contrastive Data for Better Code Retrieval and Reranking (ICLR 2025 · similarity 0.89)
- Repoformer: Selective Retrieval for Repository-Level Code Completion (ICML 2024 · similarity 0.89)
- PinNet: Pinpoint Instructive Information for Retrieval Augmented Code-to-Text Generation (ICML 2024 · similarity 0.87)
- Uncertainty-Aware Contrastive Learning with Hard Negative Sampling for Code Search Tasks (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.