Research map / Large language models / Agents, code and math
DyCodeEval: Dynamic Benchmarking of Reasoning Capabilities in Code Large Language Models Under Data Contamination
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DyCodeEval, a multi-agent pipeline that rewrites HumanEval and MBPP problems into semantically varied, complexity-preserving variants to benchmark Code LLMs under data contamination, plus a DyPass@K metric.
Problem
Static code benchmarks like HumanEval and MBPP are vulnerable to data contamination, and manual contamination-free alternatives require heavy human effort and yield variants whose complexity differs from the originals.
Approach
Applies a metamorphic relation: changing only the complexity-unrelated context of a programming problem preserves its algorithm and complexity. LLM agents (scenario proposer, context generator, prompt rewriter, validator) generate many scenario-context variants of each seed problem, so each benchmark run is fresh.
Key results
- Contaminated Llama-3.2-1B reaches Pass@3 of 0.82 on static benchmark versus 0.22 uncontaminated, while DyPass@3 drops to 0.13 on the dynamic benchmark.
- Qwen2.5-Coder-7B falls outside the 95% confidence interval of the regression on both seed datasets, suggesting contamination.
- Generated problems have external BLEU-4 of 0.17 (HumanEval) and 0.02 (MBPP) versus 0.57-0.69 for PPM, and human verification agreed on 95% of sampled pairs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Evaluate with dynamic DyPass@3 instead of static Pass@3 | Llama-3.2-1B fine-tuned on leaked data · Contaminated models | Pass@3 / DyPass@3: 0.82 -> 0.13 | Mixed |
| Evaluate with dynamic DyPass@3 instead of static Pass@3 | Llama-3.2-3B fine-tuned on leaked data · Contaminated models | Pass@3 / DyPass@3: 0.88 -> 0.24 | Mixed |
| Evaluate with dynamic DyPass@3 instead of static Pass@3 | CodeLlama-7b-hf · In-the-wild models | Pass@3 / DyPass@3: 0.39 -> 0.34 | Mixed |
| Evaluate with dynamic DyPass@3 instead of static Pass@3 | CodeLlama-13b-hf · In-the-wild models | Pass@3 / DyPass@3: 0.48 -> 0.37 | Mixed |
| Evaluate with dynamic DyPass@3 instead of static Pass@3 | Llama-3.2-1B · In-the-wild models | Pass@3 / DyPass@3: 0.22 -> 0.17 | Mixed |
| Evaluate with dynamic DyPass@3 instead of static Pass@3 | Llama-3.2-3B · In-the-wild models | Pass@3 / DyPass@3: 0.35 -> 0.31 | Mixed |
| Evaluate with dynamic DyPass@3 instead of static Pass@3 | Llama-3.1-8B · In-the-wild models | Pass@3 / DyPass@3: 0.48 -> 0.39 | Mixed |
| Evaluate with dynamic DyPass@3 instead of static Pass@3 | Llama-3.1-8B-Instruct · In-the-wild models | Pass@3 / DyPass@3: 0.72 -> 0.64 | Mixed |
| Use Claude-3.5-Haiku instead of Claude-3.5-Sonnet as the foundation model | DyCodeEval · Generated problem consistency | consistency rate: 95% -> 83% | Hurt |
| Generate variants with LLM-proposed scenarios and contexts rather than rule-based mutations | Problem diversity on HumanEval and MBPP · HumanEval and MBPP | BLEU-4 and SemSim: HumanEval internal: 1.00 -> 0.27, 1.00 -> 0.74; MBPP internal: 1.00 -> 0.18, 1.00 -> 0.73 | Helped |
Surprising findings
- Replacing Claude-3.5-Sonnet with Claude-3.5-Haiku drops the problem consistency rate from 95% to 83%.
- Contaminated DeepSeek-Coder scored below the unmodified model when fine-tuned on only 25% of the benchmark data.
Limitations
- Generation relies on a very large LLM for a high consistency rate, making it computationally costly.
- Generated prompts sometimes include excessive information that may confuse readers.
- Validation relies on LLM judges with only a small-scale human check (60 pairs).
Experimental setup
Datasets: HumanEval MBPP-Sanitized
Benchmarks: HumanEval MBPP-Sanitized DyCodeEval-generated variants
Metrics: Pass@1 Pass@K DyPass@K BLEU-4 cosine similarity of GPT-2 embeddings consistency rate
Baselines: PPM Token Mutation Char Mutation Func Mutation Insert Line CommSyntax Pass@K
Builds on: metamorphic testing DyVal PPM EvalPlus
Closest papers by meaning
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (ICLR 2025 · similarity 0.92)
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks (ICLR 2024 · similarity 0.91)
- StackEval: Benchmarking LLMs in Coding Assistance (NeurIPS 2024 · similarity 0.91)
- CodeMMLU: A Multi-Task Benchmark for Assessing Code Understanding & Reasoning Capabilities of CodeLLMs (ICLR 2025 · similarity 0.91)
- CoRe: Benchmarking LLMs’ Code Reasoning Capabilities through Static Analysis Tasks (NeurIPS 2025 · similarity 0.90)
- Unsupervised Evaluation of Code LLMs with Round-Trip Correctness (ICML 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.