Research map / Large language models / Retrieval and knowledge

CoRNStack: High-Quality Contrastive Data for Better Code Retrieval and Reranking

ICLR 2025 · Large language models › Retrieval and knowledge · approach: Retrieval-augmented reasoning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Builds CoRNStack, a 21M-example contrastive code dataset from The Stack v2 using dual consistency filtering and curriculum softmax hard-negative mining, then trains a 137M code retriever (CodeRankEmbed) and a 7B listwise code reranker (CodeRankLLM) for code retrieval and GitHub function localization.

Problem

Code embedding models trained on noisy, unfiltered (text, code) pairs with few hard negatives underperform on real-world code retrieval such as GitHub issue function localization, and code reranking is largely unexplored.

Approach

Filter heuristically mined docstring-code pairs with dual consistency filtering (top-k plus similarity threshold using a proxy embedding model), mine hard negatives offline with false-negative removal, sample them online via a temperature-annealed softmax curriculum, and distill listwise reranking supervision from a 32B teacher into a 7B code reranker.

Key results

• CodeRankEmbed (137M) reaches 77.9 average MRR@1000 on CodeSearchNet and 59.5 on AdvTest, beating CodeSage-Large (1.3B, 71.2) and other baselines.
• The 7B CodeRankLLM reranker raises CodeSearchNet average MRR@100 from 77.7 (retriever only) to 80.5 and AdvTest Python from 56.9 to 69.1.
• On SWE-Bench-Lite, CodeRankEmbed+CodeRankLLM reaches 68.2 top-1 file and 67.5 top-5 function localization, versus 44.5 top-5 for Agentless GPT-4o.

Measured effects reported in the paper

Surprising findings

• Text-finetuned Qwen-2.5-7B-Instruct reranker (78.8 CSN avg) outperforms the zero-shot code LLM reranker, and the code-finetuned reranker is needed to reach the best results.
• GPT-4o outperforms the retriever on file-level localization on SWE-Bench-Lite, possibly due to pretraining exposure to the repositories.

Limitations

• Heuristic filtering removed exact matches with evaluation sets but does not catch semantically similar queries and code (possible benchmark leakage).
• Functional correctness of code pairs is not validated by test execution due to cost.
• Function localization gains are shown only on Python SWE-Bench-Lite; the reranker's teacher-student gap remains on AdvTest.

Experimental setup

Datasets: CoRNStack The Stack v2 CodeSearchNet CosQA SWE-Bench-Lite MS MARCO

Benchmarks: CodeSearchNet AdvTest CoIR SWE-Bench-Lite

Metrics: MRR@1000 nDCG@10 MRR@100 top-k localization accuracy

Baselines: CodeSage CodeT5+ Jina-Code-v2 E5-Mistral OpenAI-Ada-002 Voyage-Code-002 Agentless Qwen-2.5-Coder-7B-Instruct zero-shot

Builds on: Arctic-Embed-M Jina-Code-v2 The Stack v2 contrastors

Closest papers by meaning

• Towards A Generalist Code Embedding Model Based On Massive Data Synthesis (NeurIPS 2025 · similarity 0.89)
• Repoformer: Selective Retrieval for Repository-Level Code Completion (ICML 2024 · similarity 0.87)
• CODE REPRESENTATION LEARNING AT SCALE (ICLR 2024 · similarity 0.87)
• CPRet: A Dataset, Benchmark, and Model for Retrieval in Competitive Programming (NeurIPS 2025 · similarity 0.87)
• CONSIDER: Commonalities and Specialties Driven Multilingual Code Retrieval Framework (AAAI 2024 · similarity 0.86)
• CodeMMR: Bridging Natural Language, Code, and Image for Unified Retrieval (CVPR 2026 · similarity 0.86)