Research map / Large language models / Agents, code and math
CPRet: A Dataset, Benchmark, and Model for Retrieval in Competitive Programming
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces CPRet, a retrieval benchmark suite and two embedding models (CPRetriever-Code with Group-InfoNCE, CPRetriever-Prob) for competitive programming problem and code retrieval, and shows high-similarity problems inflate LiveCodeBench pass rates.
Problem
Duplicate or highly similar competitive programming problems threaten contest fairness and the validity of LLM benchmarks, and existing retrieval benchmarks lack problem-level similarity tasks and strict temporal separation.
Approach
Four retrieval tasks (two code-centric, two new problem-centric: Problem-to-Duplicate and Simplified-to-Full) built from crawled problem-solution data and curated annotations, plus a Group-InfoNCE loss that treats all correct solutions of a problem as one positive group with a variance penalty, and a triplet-loss problem-level fine-tuning stage.
Key results
- CPRetriever-Prob-Qwen3-4B reaches 84.60 average NDCG@10 across the four tasks, the best in the table, with 74.33 on Problem-to-Duplicate and 96.15 on Simplified-to-Full.
- CPRetriever-Code-Qwen3-4B scores 86.22 on Text-to-Code, versus 66.62 for the base Qwen3-Embedding-4B.
- On 388 post-2024.9 LiveCodeBench problems, pass rate rises with maximum similarity to prior CPRet problems, and o3-mini/o4-mini High-vs-Low gap narrows at high similarity.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune CPRetriever-Code on problem-level retrieval tasks (ablation) | CPRetriever-Code · Problem-to-Duplicate | NDCG@10: 38.68 -> 60.06 | Helped |
| fine-tune CPRetriever-Code on problem-level retrieval tasks (ablation) | CPRetriever-Code · Text-to-Code | NDCG@10: 70.40 -> 56.50 | Hurt |
| add data augmentation during Group-InfoNCE training (ablation) | Group-InfoNCE · Text-to-Code | NDCG@10: 69.00 -> 70.11 | Helped |
| add data augmentation during Group-InfoNCE training (ablation) | Group-InfoNCE · Code-to-Code | NDCG@10: 69.18 -> 70.42 | Helped |
| replace single-positive InfoNCE with multi-positive InfoNCE (ablation) | contrastive pretraining · Text-to-Code | NDCG@10: 68.84 -> 66.44 | Hurt |
| replace single-positive InfoNCE with multi-positive InfoNCE (ablation) | contrastive pretraining · Code-to-Code | NDCG@10: 68.97 -> 66.62 | Hurt |
| increase Group-InfoNCE positive count from 4 to 16 (ablation) | Group-InfoNCE · Text-to-Code | NDCG@10: 69.00 -> 69.47 | Helped |
| increase Group-InfoNCE positive count from 4 to 16 (ablation) | Group-InfoNCE · Code-to-Code | NDCG@10: 69.18 -> 69.77 | Helped |
| add contrastive pretraining on CPRet-PCPCD (ablation) | retrieval model · Text-to-Code | NDCG@10: 39.60 -> 70.40 | Helped |
| fine-tune only on Simplified-to-Full data rather than both problem-level datasets (ablation) | contrastively pretrained model · Average of four retrieval tasks | NDCG@10: 69.50 -> 72.62 | Helped |
| add Problem-to-Duplicate data to Simplified-to-Full fine-tuning (ablation) | contrastively pretrained model · Problem-to-Duplicate | NDCG@10: 57.01 -> 60.06 | Helped |
| add CPRet-PCPCD data during problem-level fine-tuning (ablation) | contrastively pretrained model · Text-to-Code | NDCG@10: 68.05 -> 71.15 | Helped |
Surprising findings
- Problem-level fine-tuning lowers Text-to-Code performance (e.g., 68.54 to 65.85 for Qwen3-4B), a trade-off between code-level and problem-level objectives.
- Weaker models benefit more from high-similarity problems, so performance gaps between models shrink on such problems.
Limitations
- Test sets for code-centric tasks may leak into future training corpora, so benchmark scores may inflate over time.
- Only about 700 duplicate pairs are annotated, and many duplicates likely remain undiscovered.
- Analysis is limited to competitive programming; math competitions are not studied.
Experimental setup
Datasets: CPRet-PCPCD APPS CodeContests TACO Codeforces submissions CodeChef submissions Luogu simplified statements
Benchmarks: CPRet Text-to-Code CPRet Code-to-Code CPRet Problem-to-Duplicate CPRet Simplified-to-Full LiveCodeBench
Metrics: NDCG@10 pass rate Pass@1
Baselines: SFR-Embedding-Code-2B Qwen3-Embedding-4B Qodo-Embed-1-7B Qwen3-Embedding-8B CodeSage-large-v2 gte-modernbert-base
Builds on: InfoNCE SFR-Embedding-Code-2B Qwen3-Embedding-4B CoIR
Closest papers by meaning
- rStar-Coder: Scaling Competitive Code Reasoning with a Large-Scale Verified Dataset (NeurIPS 2025 · similarity 0.88)
- ICPC-Eval: Probing the Frontiers of LLM Reasoning with Competitive Programming Contests (NeurIPS 2025 · similarity 0.87)
- Simple Baselines are Competitive with Code Evolution (NeurIPS 2026 · similarity 0.87)
- LiveCodeBench Pro: How Do Olympiad Medalists Judge LLMs in Competitive Programming? (NeurIPS 2025 · similarity 0.87)
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (ICLR 2025 · similarity 0.87)
- CoRNStack: High-Quality Contrastive Data for Better Code Retrieval and Reranking (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.