Research map / Large language models / Agents, code and math

ConvCodeWorld: Benchmarking Conversational Code Generation in Reproducible Feedback Environments

ICLR 2025 · Large language models › Agents, code and math · approach: Software engineering agents · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces ConvCodeWorld, a reproducible multi-turn code generation environment with nine feedback combinations (compilation, partial/full execution, GPT-4o verbal feedback), plus ConvCodeWorld-Static using pre-generated logs from CodeLlama-7B.

Problem

Existing code generation benchmarks are mostly single-turn and existing multi-turn benchmarks (InterCode, MINT) lack diversity of feedback types and combinations, and live verbal feedback generation with LLMs is costly and hard to reproduce.

Approach

Systematically combine three feedback types (compilation, execution with partial/full test coverage, novice/expert GPT-4o verbal feedback) into nine scenarios over BigCodeBench, and build a static benchmark from pre-generated logs of a weak reference model that correlates with the live environment (Spearman 0.82-0.99).

Key results

• Spearman rank correlations between ConvCodeWorld-Static and live ConvCodeWorld range from 0.82 to 0.99 using CodeLlama-7B-Instruct logs as reference.
• With expert verbal feedback, DeepSeek-Coder-6.7B-Instruct reaches 82.8 Recall, above GPT-4o's 82.3 in the <f_c, phi, f_v*> setting.
• GPT-4o achieves the highest MRR (50.8 single-turn) while GPT-4-0613 attains the highest Recall, showing MRR and Recall trade-offs.

Measured effects reported in the paper

Surprising findings

• Training ReflectionCoder on one feedback combination reduces performance on other combinations; with expert feedback ReflectionCoder-DS-33B scores below its base DeepSeek-Coder-33B-Instruct.
• Some weaker models (e.g., Qwen1.5-32B-Chat) perform worse when execution feedback is added to expert verbal feedback.

Limitations

• Verbal feedback is simulated by GPT-4o rather than real humans; human evaluation shows expert feedback is less human-like than ShareGPT logs.
• Built on a single Python benchmark (BigCodeBench) with a fixed 10-turn horizon.
• Static benchmark relies on a single weak reference model (CodeLlama-7B), which may bias rankings.

Experimental setup

Datasets: BigCodeBench-Full-Instruct

Benchmarks: ConvCodeWorld ConvCodeWorld-Static BigCodeBench-Full-Instruct

Metrics: Pass@1 MRR Recall Spearman rank correlation

Baselines: InterCode MINT GPT-4-0613 GPT-4-Turbo-2024-04-09 GPT-4o-2024-05-13 DeepSeek-Coder-6.7B-Instruct ReflectionCoder-DS-33B

Builds on: BigCodeBench DSPy InterCode MINT

Closest papers by meaning

• MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback (ICLR 2024 · similarity 0.89)
• What Makes Large Language Models Reason in (Multi-Turn) Code Generation? (ICLR 2025 · similarity 0.89)
• Rethinking Verification for LLM Code Generation: From Generation to Testing (NeurIPS 2025 · similarity 0.88)
• Teaching Language Models to Critique via Reinforcement Learning (ICML 2025 · similarity 0.88)
• How efficient is LLM-generated code? A rigorous & high-standard benchmark (ICLR 2025 · similarity 0.88)
• INDICT: Code Generation with Internal Dialogues of Critiques for Both Security and Helpfulness (NeurIPS 2024 · similarity 0.88)