Research map / Large language models / Agents, code and math
LLMDFA: Analyzing Dataflow in Code with Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LLMDFA, a compilation-free, customizable dataflow analysis that uses LLMs to synthesize extractor and Z3 scripts and few-shot CoT summaries to detect bugs in Java programs.
Problem
Dataflow analysis traditionally requires successful compilation and expert customization, which limits applicability to incomplete programs and usability for evolving analysis needs; direct LLM-based analysis suffers hallucinations.
Approach
Decompose dataflow analysis into source/sink extraction, few-shot chain-of-thought dataflow summarization, and path feasibility validation, with LLMs synthesizing scripts that call external tools (tree-sitter parsing, Z3 SMT solver) to offload delicate reasoning.
Key results
- With gpt-3.5, LLMDFA achieves 73.75%/100%/100% precision and 92.16%/92.31%/78.38% recall for DBZ/XSS/OSCI detection on Juliet.
- On TaintBench Android malware apps (gpt-3.5, subset of 80 paths), LLMDFA reaches 75.38% precision and 61.25% recall (F1 0.67), vs 43.48%/25.00% for end-to-end LLM analysis and 72.92%/43.75% for CodeFuseQuery.
- Average LLMDFA F1 improves by 0.23 over CodeFuseQuery and 0.36 over end-to-end few-shot CoT on Juliet.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LLMDFA instead of CodeFuseQuery | Juliet DBZ detection · Juliet DBZ | F1: +0.23 F1 | Helped |
| Use LLMDFA instead of end-to-end few-shot CoT analysis | Juliet bug detection · Juliet DBZ, XSS, and OSCI | F1: +0.36 F1 | Helped |
| Use LLMDFA instead of end-to-end analysis | TaintBench · TaintBench | F1: +0.35 F1 | Helped |
| Add synthesized source/sink extractors instead of directly prompting the LLM (ablation) | LLMDFA · Juliet DBZ, XSS, and OSCI | precision | Helped |
| Add few-shot CoT prompting for dataflow summarization (ablation) | LLMDFA · Juliet DBZ | F1 | Helped |
| Add few-shot CoT prompting for dataflow summarization (ablation) | LLMDFA · Juliet DBZ | precision: 86.8% -> 73.7% | Hurt |
| Synthesize programs invoking SMT solvers for path validation instead of directly prompting the LLM (ablation) | LLMDFA · Juliet XSS and OSCI | bug detection performance | No clear effect |
Surprising findings
- LLMs often mis-encode library-based branch conditions, e.g., interpreting Math.abs(b) > 1 as And instead of Or, yet wrong encodings still often refute infeasible paths.
- Pinpoint, which relies on LLVM IR, achieves high DBZ precision (93.78%) but low recall, while the compilation-free approach has lower precision on DBZ.
Limitations
- Few-shot CoT prompts are lengthy, causing high token cost and latency, making whole-program analysis expensive.
- Summarization can be imprecise for large functions or complex pointer operations, and path conditions may be encoded incorrectly, compromising soundness.
- Cannot retrieve user-defined functions or track global variables modified at multiple locations, causing false positives or negatives.
Experimental setup
Datasets: Juliet Test Suite TaintBench SecBench.js
Benchmarks: Juliet Test Suite TaintBench SecBench.js Juliet Test Suite C/C++
Metrics: precision recall F1 score
Baselines: CodeFuseQuery Pinpoint LLM-based end-to-end analysis NoSynExt NoCoT NoSynVal
Builds on: LLVM Z3 tree-sitter few-shot chain-of-thought prompting
Closest papers by meaning
- CoRe: Benchmarking LLMs’ Code Reasoning Capabilities through Static Analysis Tasks (NeurIPS 2025 · similarity 0.86)
- RepoAudit: An Autonomous LLM-Agent for Repository-Level Code Auditing (ICML 2025 · similarity 0.85)
- CodeMMLU: A Multi-Task Benchmark for Assessing Code Understanding & Reasoning Capabilities of CodeLLMs (ICLR 2025 · similarity 0.84)
- LeDex: Training LLMs to Better Self-Debug and Explain Code (NeurIPS 2024 · similarity 0.84)
- IRIS: LLM-Assisted Static Analysis for Detecting Security Vulnerabilities (ICLR 2025 · similarity 0.84)
- BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.