Research map / Large language models / Agents, code and math
Interactive Evaluation of Large Language Models for Multi-Requirement Software Engineering Tasks
Dimitrios Rontogiannis, Maxime Peyrard, Nicolas Baldwin, Martin Josifoski, Robert West, Dimitrios Gunopulos
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| provide interviewer hints to the interviewee during interactive evaluation | o4-mini · DevAI interactive evaluation (GPT-4.1-mini interviewer) | avg requirements passed | Improved |
| provide hints to GPT-4.1-mini interviewee | GPT-4.1-mini · DevAI interactive evaluation | avg requirements passed | No clear effect |
| use GPT-4o-mini as interviewer instead of GPT-4.1-mini (ablation) | interviewee models · DevAI interactive evaluation | avg requirements passed | Hurt |
| add interviewer hints for GPT-3.5-turbo | GPT-3.5-turbo · DevAI interactive evaluation | avg requirements passed | No clear effect |
| add hints with ground truth references | all interviewees · DevAI, Dataset or Environment requirement category | requirements fixed | Improved |
| dependency-aware interactive evaluation vs static benchmark ranking | model ranking · DevAI vs published static benchmarks | rank agreement | Mixed |
Closest papers by meaning
- ConvCodeWorld: Benchmarking Conversational Code Generation in Reproducible Feedback Environments (ICLR 2025 · similarity 0.88)
- Copilot Arena: A Platform for Code LLM Evaluation in the Wild (ICML 2025 · similarity 0.87)
- CodeMMLU: A Multi-Task Benchmark for Assessing Code Understanding & Reasoning Capabilities of CodeLLMs (ICLR 2025 · similarity 0.87)
- Code Graph Model (CGM): A Graph-Integrated Large Language Model for Repository-Level Software Engineering Tasks (NeurIPS 2025 · similarity 0.87)
- CoRe: Benchmarking LLMs’ Code Reasoning Capabilities through Static Analysis Tasks (NeurIPS 2025 · similarity 0.87)
- Unsupervised Evaluation of Code LLMs with Round-Trip Correctness (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
