Research map / Large language models / Agents, code and math
CoRe: Benchmarking LLMs’ Code Reasoning Capabilities through Static Analysis Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces CoRe, a human-verified multilingual benchmark of 12,553 static-analysis task instances (data dependency, control dependency, information flow) over C/C++, Java, and Python, and evaluates 10 LLMs on it.
Problem
Existing code benchmarks evaluate end-to-end outcomes (repair, generation) and do not directly test LLMs' fine-grained reasoning about program semantics such as value propagation and control dependencies.
Approach
A semantics-aware diverse sampling strategy selects targets and task instances by structural coverage and dependency depth, and a semi-automated, human-verified annotation pipeline produces dependency labels for intra-procedural queries (pairwise and target-centric) across three languages.
Key results
- Gemini 2.5 Pro achieves the best overall results on CoRe Lite, with 91.74% overall classification F1, 84.02% correct trace rate, and 50.25% exact match enumeration.
- Most models achieve 68.9%-92.56% F1 on dependency classification, while trace generation and source enumeration fall below 50% for many models.
- Reasoning models outperform non-reasoning models by 5.2-31.5%; performance drops up to 49.5% for reverse (backward) dependencies.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a clipped prompt containing only the target function instead of the whole file (ablation) | LLMs evaluated on CoRe · CoRe Lite pairwise tasks | performance: up to 5.7% | Helped |
| Add retrieved few-shot examples to the base prompt (ablation) | LLMs evaluated on CoRe · CoRe Lite | performance | Hurt |
| Ask for a supporting trace along with dependency classification instead of classification only (ablation) | LLMs evaluated on CoRe · CoRe pairwise tasks | recall | Helped |
| Ask for a supporting trace along with dependency classification instead of classification only (ablation) | LLMs evaluated on CoRe · CoRe pairwise tasks | precision | Hurt |
| Increase function length from short (21--40 LoC) to long (81--100 LoC) | LLMs evaluated on CoRe · CoRe trace generation | Correct Trace Rate: 8.9--15.8% | Hurt |
| Increase the number of conditions in a function | LLMs evaluated on CoRe · CoRe trace generation | Correct Trace Rate: 24.4--44.6% | Hurt |
| Include early-exit logic that influences the target variable | LLMs evaluated on CoRe · CoRe source enumeration | Exact Match: up to 21.7% | Hurt |
| Use a reverse dependency where the source appears after the target | LLMs evaluated on CoRe · CoRe pairwise classification | F1: up to 49.5% | Hurt |
| Use reasoning-capable models rather than non-reasoning models | LLMs evaluated on CoRe · CoRe tasks | performance: 5.2--31.5% | Helped |
Surprising findings
- Providing the entire file instead of only the target function degrades performance by up to 5.7%, and few-shot retrieved examples do not improve and can degrade results.
- When asked to produce traces, models over-predict dependency existence, yielding higher precision but lower recall in the classification-only setting comparison.
Limitations
- Restricted to intra-procedural analysis on short functions (within 100 LoC).
- Annotation is semi-automated and manual, so may contain errors; inter-agreement is 87.5%.
- Evaluates only ten models, mostly large API models, with limited prompt and retrieval variants.
Experimental setup
Datasets: CodeNet Google Code Jam
Benchmarks: CoRe CoRe Lite
Metrics: F1 precision recall correct trace rate exact match
Baselines: Claude 3.7 Claude 3.5 DeepSeek R1 DeepSeek V3 Gemini 2.5 Pro GPT o3 GPT o4-mini GPT 4o Llama 3.1 405B Qwen3 235B
Builds on: CodeNet CRUXEval LLMDFA IRIS
Closest papers by meaning
- Evaluating Program Semantics Reasoning with Type Inference in System $F$ (NeurIPS 2025 · similarity 0.91)
- SemCoder: Training Code Language Models with Comprehensive Semantics Reasoning (NeurIPS 2024 · similarity 0.91)
- CodeMMLU: A Multi-Task Benchmark for Assessing Code Understanding & Reasoning Capabilities of CodeLLMs (ICLR 2025 · similarity 0.90)
- DyCodeEval: Dynamic Benchmarking of Reasoning Capabilities in Code Large Language Models Under Data Contamination (ICML 2025 · similarity 0.90)
- StackEval: Benchmarking LLMs in Coding Assistance (NeurIPS 2024 · similarity 0.89)
- Can Large Language Models Understand Intermediate Representations in Compilers? (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.