Research map / Large language models / Agents, code and math
Incoherence as Oracle-less Measure of Error in LLM-Based Code Generation
Thomas Jean-Michel Valentin, Ardi Madadi, Gaetano Sapia, Marcel Böhme
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| sample more programs per task (m from 10 to 50) (ablation) | Incoherence detection (GPT-4o) · MBPP | detection rate: 0.7243 -> 0.8105 | Improved |
| sample more programs per task (m from 25 to 50) (ablation) | Incoherence detection (GPT-4o) · HumanEval | detection rate: 0.8267 -> 0.8182 | No clear effect |
| sample a second program (m=2 vs m=1) (ablation) | Incoherence detection (GPT-4o) · MBPP | detection rate: 0.0000 -> 0.3974 | Improved |
| raise sampling temperature from 0.2 to 1.0 (ablation) | Incoherence detection (GPT-4o) · MBPP | detection rate: 0.5422 -> 0.8050 | Improved |
| raise sampling temperature from 0.2 to 1.0 (ablation) | Incoherence detection (GPT-4o) · HumanEval | detection rate: 0.5556 -> 0.7600 | Improved |
| generate more test inputs (n from 100 to 10000) (ablation) | Incoherence detection (GPT-4o) · MBPP | detection rate: 0.6811 -> 0.7445 | Improved |
| generate more test inputs (n from 1000 to 10000) (ablation) | Incoherence detection (GPT-4o) · HumanEval | detection rate: 0.7286 -> 0.7222 | No clear effect |
| use zero incoherence as a filter for likely-correct programs | 16 LLMs · MBPP | mean error: 0.3009 -> 0.1866 | Improved |
| use zero incoherence as a filter for likely-correct programs | 16 LLMs · HumanEval | mean error: 0.1050 -> 0.0471 | Improved |
| rank LLMs by incoherence instead of by pass@1 | 16 LLMs · HumanEval | Spearman rho of rankings: rho = 0.94 | Improved |
Closest papers by meaning
- How efficient is LLM-generated code? A rigorous & high-standard benchmark (ICLR 2025 · similarity 0.90)
- Can LLM Replace Stack Overflow? A Study on Robustness and Reliability of Large Language Model Code Generation (AAAI 2024 · similarity 0.88)
- Unsupervised Evaluation of Code LLMs with Round-Trip Correctness (ICML 2024 · similarity 0.88)
- EffiBench-X: A Multi-Language Benchmark for Measuring Efficiency of LLM-Generated Code (NeurIPS 2025 · similarity 0.88)
- LLM-Assisted Code Cleaning For Training Accurate Code Generators (ICLR 2024 · similarity 0.87)
- StackEval: Benchmarking LLMs in Coding Assistance (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
