Research map / Large language models / Agents, code and math
CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add BM25-retrieved cross-file context to prompt (ablation) | StarCoder-15.5B · CrossCodeEval Python | code EM: 8.82 -> 15.72 | Helped |
| add BM25-retrieved cross-file context to prompt (ablation) | StarCoder-15.5B · CrossCodeEval TypeScript | code ES: 51.22 -> 44.87 | Hurt |
| add BM25-retrieved cross-file context to prompt (ablation) | GPT-3.5-turbo · CrossCodeEval Java | code EM: 12.30 -> 19.12 | Helped |
| add BM25-retrieved cross-file context to prompt (ablation) | CodeGen25-7B · CrossCodeEval C# | code EM: 4.36 -> 13.01 | Helped |
| build retrieval query with reference completion (oracle) (ablation) | StarCoder-15.5B · CrossCodeEval Python | code EM: 15.72 -> 21.01 | Helped |
| replace BM25 with OpenAI ada embeddings for retrieval (ablation) | StarCoder-15.5B · CrossCodeEval C# | code EM: 13.57 -> 15.39 | Helped |
| replace BM25 with UniXCoder for retrieval (ablation) | StarCoder-15.5B · CrossCodeEval C# | code EM: 13.57 -> 11.93 | Hurt |
| use nucleus sampling with reranking instead of greedy (ablation) | StarCoder-15.5B · CrossCodeEval Python | code EM: 8.82 -> 8.93 | No clear effect |
| scale model size (CodeGen-350M to 16.1B) | CodeGen · CrossCodeEval Python | code EM: 2.70 -> 6.87 | Helped |
Closest papers by meaning
- Hierarchical Context Pruning: Optimizing Real-World Code Completion with Repository-Level Pretrained Code LLMs (AAAI 2025 · similarity 0.89)
- Repoformer: Selective Retrieval for Repository-Level Code Completion (ICML 2024 · similarity 0.88)
- McEval: Massively Multilingual Code Evaluation (ICLR 2025 · similarity 0.87)
- RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems (ICLR 2024 · similarity 0.87)
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (ICLR 2025 · similarity 0.87)
- CoRNStack: High-Quality Contrastive Data for Better Code Retrieval and Reranking (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.