Research map / Large language models / Agents, code and math
Unsupervised Evaluation of Code LLMs with Round-Trip Correctness
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes round-trip correctness (RTC), an unsupervised evaluation where an LLM describes code and then regenerates it from the description, scored by unit tests or exact match, and shows it correlates with pass@1 on HumanEval and ARCADE while scaling to 58 real-world projects and code editing.
Problem
Code LLM evaluation relies on a few small, manually curated benchmarks (HumanEval, MBPP, ARCADE, DS-1000) that cover narrow software domains and are costly to expand.
Approach
Evaluate a model by chaining a forward task (code-to-natural-language description) with its inverse backward task (description-to-code) and checking whether the regenerated code is semantically equivalent to the original, using existing unit tests or exact match as the similarity function, with a forward-lift measure against an uninformative description.
Key results
- RTC-pass correlates with pass@1 on HumanEval (Pearson r=0.96, Spearman 0.90) and ARCADE (r=0.96, Spearman 0.81) across seven models.
- Across 58 projects, Gemini Pro averages a forward lift of 21.5% versus 7.0% for Gemini Nano 2, and RTC-pass varies widely by project domain.
- On CodeReviewer code editing (1K samples), Gemini Pro achieves RTC exact match of 12.9% versus 5.2% for Gemini Nano 2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RTC_pass instead of standard pass@1 to evaluate synthesis | Code LLM evaluation · HumanEval and ARCADE | correlation with pass@1: r=0.96 for HumanEval and r=0.96 for ARCADE | Helped |
| Evaluate models across open-source project domains with RTC_pass | Gemini Nano 2 and Gemini Pro · 58 open-source Python projects | RTC_pass: r=0.75, Spearman's ρ=0.76 | Mixed |
| Use Gemini Pro instead of Gemini Nano 2 | EditingRTC · CodeReviewer test set sample (1K) | RTC_exact_match: 5.2 -> 12.9 | Helped |
| Use Gemini Pro instead of Gemini Nano 2 | EditingRTC · CodeReviewer test set sample (1K) | forward lift (ExactMatch): 4.8 -> 12.4 | Helped |
Surprising findings
- Reference-based BLEU for edit descriptions is very low and uninformative, while RTC exact match can give a perfect score for a case where BLEU against the PR comment is near zero.
- A trivial baseline that retains the input unchanged scores well on standard editing metrics such as BLEU, ROUGE and BERTScore.
Limitations
- RTC quality depends on the similarity function; weak similarity metrics may give arbitrary results.
- Forward and backward task performance are coupled, so if the forward model yields implausible descriptions the backward model cannot be measured reliably.
- A model trained to recite its input can achieve perfect RTC by copying, so the metric assumes reasonably trained instruction-tuned LLMs.
Experimental setup
Datasets: HumanEval ARCADE CodeReviewer 58 open-source Python projects (5,961 samples)
Benchmarks: HumanEval ARCADE CodeReviewer 58 open-source Python projects
Metrics: RTC pass pass@1 forward lift RTC exact match RTC BLEU RTC ROUGE Pearson correlation Spearman correlation
Baselines: pass@1 exact match BLEU ROUGE BERTScore HumanEvalExplain
Builds on: property-based testing IdentityChain back translation HumanEvalPack
Closest papers by meaning
- How efficient is LLM-generated code? A rigorous & high-standard benchmark (ICLR 2025 · similarity 0.90)
- DyCodeEval: Dynamic Benchmarking of Reasoning Capabilities in Code Large Language Models Under Data Contamination (ICML 2025 · similarity 0.90)
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (ICLR 2025 · similarity 0.89)
- CodeMMLU: A Multi-Task Benchmark for Assessing Code Understanding & Reasoning Capabilities of CodeLLMs (ICLR 2025 · similarity 0.88)
- Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks (ICML 2024 · similarity 0.88)
- EvoCodeBench: An Evolving Code Generation Benchmark with Domain-Specific Evaluations (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.