Research map / Large language models / Agents, code and math
Can Large Language Models Understand Intermediate Representations in Compilers?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Empirical study evaluating six LLMs (GPT-4, GPT-3, DeepSeek R1, Gemma 2, Llama 3.1, Code Llama) on four LLVM IR comprehension tasks (CFG reconstruction, decompilation, summarization, execution reasoning) built from HumanEval, finding strong gaps in control-flow and instruction-level reasoning.
Problem
Whether LLMs can understand and reason over compiler intermediate representations (LLVM IR), covering structural, syntactic, semantic, and execution understanding, remains underexplored.
Approach
Builds an evaluation framework that compiles HumanEval C++ programs to LLVM IR at -O0 to -O3 and probes LLMs with four IR tasks, with compiler-generated ground truth for CFGs, and compares IR execution reasoning against source-code execution reasoning.
Key results
- GPT-4 completed all 164 CFG tasks but only 39 were fully correct, while DeepSeek R1 produced 57 fully accurate CFGs from 77 completed cases.
- In execution reasoning on IR, GPT-4 achieved the highest IR pass count (59 of 164, pass rate 0.36), while source-code pass rates were substantially higher (e.g., GPT-4 0.72).
- Code Llama failed to produce a single accurate CFG (0 full accuracy) and failed on 99 decompilation applications at -O0.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DeepSeek R1 rather than GPT-4 for CFG reconstruction | LLM CFG reconstruction · HumanEval IRs | full CFG accuracy (count): 39 -> 57 | Helped |
| Apply Chain-of-Thought prompting for stepwise CFG reconstruction | LLM CFG reconstruction · HumanEval IRs | full CFG accuracy | Helped |
| Compile IR at -O0 rather than -O1, -O2, or -O3 | LLM IR decompilation · HumanEval IR | re-execution success | Mixed |
| Use DeepSeek R1 rather than GPT-4 for IR summarization | LLM code summarization · HumanEval IR summaries | BLEU: 0.35 -> 0.42 | Helped |
| Use DeepSeek R1 rather than GPT-4 for IR summarization | LLM code summarization · HumanEval IR summaries | ROUGE-L: 0.56 -> 0.64 | Helped |
| Use DeepSeek R1 rather than GPT-4 for IR summarization | LLM code summarization · HumanEval IR summaries | METEOR: 0.63 -> 0.705 | Helped |
| Provide LLVM IR rather than source code for execution reasoning | LLM execution reasoning · HumanEval | pass rate: 0.36 -> 0.72 | Hurt |
| Provide LLVM IR rather than source code for execution reasoning | LLM execution reasoning · HumanEval | pass rate: 0.195 -> 0.804 | Hurt |
| Provide LLVM IR rather than source code for execution reasoning | LLM execution reasoning · HumanEval | assertion pass rate | Hurt |
| Use chain-of-thought prompting for execution reasoning | LLMs reasoning over IR · HumanEval IR execution reasoning | execution correctness | No clear effect |
Surprising findings
- Models performed markedly better on equivalent source code than on LLVM IR for execution reasoning, despite IR being a lower-level form.
- Re-execution success peaked at -O1 and -O2 rather than -O0, and -O3 degraded decompilation accuracy.
Limitations
- Evaluation limited to six models and HumanEval-derived short programs, which may not reflect real-world IR diversity.
- Only expert meta-prompts were used; advanced prompting techniques were not explored.
- Several models hit context-window limits at -O0, causing task failures.
Experimental setup
Datasets: HumanEval
Benchmarks: HumanEval-derived LLVM IR benchmark
Metrics: task completion CFG node accuracy full CFG accuracy partial CFG accuracy re-execution success BLEU ROUGE-L METEOR assertion pass rate
Baselines: NCC IR2Vec FAIR
Builds on: HumanEval LLVM chain-of-thought prompting
Closest papers by meaning
- CoRe: Benchmarking LLMs’ Code Reasoning Capabilities through Static Analysis Tasks (NeurIPS 2025 · similarity 0.88)
- SemCoder: Training Code Language Models with Comprehensive Semantics Reasoning (NeurIPS 2024 · similarity 0.88)
- Chain of Execution Supervision Promotes General Reasoning in Large Language Models (NeurIPS 2025 · similarity 0.88)
- REASONING COMPILER: LLM-Guided Optimizations for Efficient Model Serving (NeurIPS 2025 · similarity 0.87)
- AST-T5: Structure-Aware Pretraining for Code Generation and Understanding (ICML 2024 · similarity 0.87)
- Can LLMs Separate Instructions From Data? And What Do We Even Mean By That? (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.