Research map / Large language models / Agents, code and math
CodeHalu: Investigating Code Hallucinations in LLMs via Execution-based Verification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Defines code hallucinations via execution-based verification, proposes the CodeHalu detection algorithm and the CodeHaluEval benchmark (8,883 samples, 699 tasks from APPS), and evaluates 17 LLMs on hallucination rates across four categories.
Problem
LLM-generated code is often syntactically correct and semantically plausible yet fails to execute as expected or meet requirements, and this code-domain hallucination phenomenon has not been systematically studied or quantified.
Approach
Defines code hallucinations through execution verification, uses a cross-model statistical induction algorithm (CodeHalu) to find recurring failure patterns across many LLMs, and classifies them into mapping, naming, resource, and logic hallucinations with eight subcategories.
Key results
- Across 17 evaluated LLMs, average hallucination rates range from roughly 20% to 60%, with logic hallucinations the most prevalent category.
- GPT-4 has the lowest average hallucination rate in the table (33.04), while Gemma-7B has the highest (61.53).
- Cross-task occurrence rate averages only 2.04%, which the authors use to argue the eight categories are independent.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GPT-4 rather than Gemma-7B | Code generation models evaluated with CodeHaluEval · CodeHaluEval | average hallucination rate: 61.53 -> 33.04 | Helped |
| Use GPT-4 rather than GPT-3.5 | Code generation models evaluated with CodeHaluEval · CodeHaluEval — Data Compliance | hallucination rate: 20.19 -> 32.31 | Hurt |
| Use GPT-4 rather than GPT-3.5 | Code generation models evaluated with CodeHaluEval · CodeHaluEval — Structure Access | hallucination rate: 22.05 -> 10.02 | Helped |
| Use Claude-3-haiku rather than GPT-4 | Code generation models evaluated with CodeHaluEval · CodeHaluEval — Identity | hallucination rate: 27.74 -> 9.07 | Helped |
| Use LLaMA-3-8B rather than GPT-4 | Code generation models evaluated with CodeHaluEval · CodeHaluEval — Logic Deviation | hallucination rate: 85.76 -> 78.39 | Helped |
| Use GPT-4 rather than Gemma-7B | Code generation models evaluated with CodeHaluEval · CodeHaluEval — Logic Breakdown | hallucination rate: 100.00 -> 0.51 | Helped |
| Use GPT-4 rather than LLaMA-3-8B | Code generation models evaluated with CodeHaluEval · CodeHaluEval | average hallucination rate: 33.67 -> 33.04 | No clear effect |
Surprising findings
- Logic hallucinations dominate across nearly all models, and naming and resource hallucinations are comparatively rare.
- Only 9 of 16 models occasionally produced syntax errors, with an average error rate of 0.0020 in a preliminary test.
Limitations
- Focuses only on Python, so findings may not transfer to other languages.
- Security risks are out of scope and are assumed to be handled by sandboxing.
- Taxonomy validated by a small questionnaire (16 valid responses) and team-based heuristic classification.
Experimental setup
Datasets: APPS
Benchmarks: CodeHaluEval
Metrics: Hallucination Rate (HR)
Baselines: GPT-4 GPT-3.5 LLaMA-3-8B DeepSeek Coder-6.7B Claude-3-haiku Code LLaMA-7B StarCoder-16B WizardCoder-7B
Builds on: APPS HumanEval
Closest papers by meaning
- LLM-Check: Investigating Detection of Hallucinations in Large Language Models (NeurIPS 2024 · similarity 0.90)
- Scalable Token-Level Hallucination Detection in Large Language Models (NeurIPS 2026 · similarity 0.89)
- CodeMMLU: A Multi-Task Benchmark for Assessing Code Understanding & Reasoning Capabilities of CodeLLMs (ICLR 2025 · similarity 0.88)
- Why and How LLMs Hallucinate: Connecting the Dots with Subsequence Associations (NeurIPS 2025 · similarity 0.87)
- HaDeMiF: Hallucination Detection and Mitigation in Large Language Models (ICLR 2025 · similarity 0.87)
- Reducing Tool Hallucination via Reliability Alignment (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.