Research map / Large language models / Language, safety and interpretability
Multi-lingual Evaluation of Code Generation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Provide the Python canonical solution as translation context | 13B multilingual model · MathQA-Java | pass@100 execution score (%): 18.05 -> 96.33 | Helped |
| Provide the Python canonical solution as translation context | 13B multilingual model · MathQA-JS | pass@100 execution score (%): 10.67 -> 98.08 | Helped |
| Prepend four MathQA examples with canonical solutions | 13B multilingual model · MathQA-Python | pass@100 execution score (%): 13.43 -> 21.50 | Helped |
| Prepend four MathQA examples with canonical solutions | 13B multilingual model · MathQA-Java | pass@100 execution score (%): 18.05 -> 26.21 | Helped |
| Prepend four MathQA examples with canonical solutions | 13B multilingual model · MathQA-JS | pass@100 execution score (%): 10.67 -> 24.96 | Helped |
| Use multilingual rather than monolingual training | 13B code model · MathQA | pass@100 execution score (%): Python: 20.23 -> 13.43; Java: 14.86 -> 18.05; JS: 10.78 -> 10.67 | Mixed |
| Add right context to code insertion | InCoder · i-MBPP | Pass@1 accuracy: 30.1 -> 37.07; 23.2% relative improvement | Helped |
| Add right context to code insertion | InCoder · i-MBJSP | Pass@1 accuracy: 48.65 -> 55.68; 14.4% relative improvement | Helped |
| Add right context to code insertion | InCoder · i-MBJP | Pass@1 accuracy: 41.7 -> 57.41; 37.6% relative improvement | Helped |
| Increase right context from 0 to 1 line (ablation) | InCoder · i-MBPP | Pass@1 accuracy: 30.1 -> 32.1 | Helped |
| Increase right context from 1 to 2 lines (ablation) | InCoder · i-MBPP | Pass@1 accuracy: 32.1 -> 35.6 | Helped |
| Increase right context from 2 to 3 lines (ablation) | InCoder · i-MBPP | Pass@1 accuracy: 35.6 -> 36.4 | Helped |
Closest papers by meaning
- McEval: Massively Multilingual Code Evaluation (ICLR 2025 · similarity 0.90)
- Automatically Testing Functional Properties of Code Translation Models (AAAI 2024 · similarity 0.87)
- What Makes Large Language Models Reason in (Multi-Turn) Code Generation? (ICLR 2025 · similarity 0.87)
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (ICLR 2025 · similarity 0.86)
- Teaching Large Language Models to Self-Debug (ICLR 2024 · similarity 0.86)
- EvoCodeBench: An Evolving Code Generation Benchmark with Domain-Specific Evaluations (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.